[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:00:10 dist-test-slave-jnhs krb5kdc[8607](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:00:10 dist-test-slave-jnhs krb5kdc[8607](info): set up 2 sockets
Sep 26 15:00:10 dist-test-slave-jnhs krb5kdc[8607](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260926 15:00:11.431313 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:00:11 dist-test-slave-jnhs krb5kdc[8607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:00:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:00:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:00:11.559825 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:36385
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:36385
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:11.642778 8620 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:11.643004 8620 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:11.643030 8620 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:11.644250 8620 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:11.644302 8620 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:11.644316 8620 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:11.644330 8620 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:11.644342 8620 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:00:11.645923 8620 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:36385
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:36385
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8620
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:11.646215 8620 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:11.646471 8620 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:11.651496 8625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:11.651496 8628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:11.651726 8620 server_base.cc:1034] running on GCE node
W20260926 15:00:11.651554 8626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:11.651988 8620 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:11.652486 8620 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:11.653689 8620 hybrid_clock.cc:648] HybridClock initialized: now 1790434811653673 us; error 48 us; skew 500 ppm
Sep 26 15:00:11 dist-test-slave-jnhs krb5kdc[8607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434811, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:11.656483 8620 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:00:11.656948 8620 webserver.cc:460] Webserver started at http://127.8.102.62:37995/ using document root <none> and password file <none>
I20260926 15:00:11.657189 8620 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:11.657258 8620 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:11.657414 8620 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:11.658509 8620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "9821273ab0634f009dbb0e6c6d30da99"
format_stamp: "Formatted at 2026-09-26 15:00:11 on dist-test-slave-jnhs"
server_key: "c2125d4e152aa9113267ed935ee0dc1b"
server_key_iv: "a8f6f6e7fa33fde2f174755a1a997df8"
server_key_version: "encryptionkey@0"
I20260926 15:00:11.658968 8620 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "9821273ab0634f009dbb0e6c6d30da99"
format_stamp: "Formatted at 2026-09-26 15:00:11 on dist-test-slave-jnhs"
server_key: "c2125d4e152aa9113267ed935ee0dc1b"
server_key_iv: "a8f6f6e7fa33fde2f174755a1a997df8"
server_key_version: "encryptionkey@0"
I20260926 15:00:11.660403 8620 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:11.661213 8635 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:11.661396 8620 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:11.661509 8620 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "9821273ab0634f009dbb0e6c6d30da99"
format_stamp: "Formatted at 2026-09-26 15:00:11 on dist-test-slave-jnhs"
server_key: "c2125d4e152aa9113267ed935ee0dc1b"
server_key_iv: "a8f6f6e7fa33fde2f174755a1a997df8"
server_key_version: "encryptionkey@0"
I20260926 15:00:11.661592 8620 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:11.671360 8620 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:11.673765 8620 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:11.673940 8620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:11.678083 8620 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:36385
I20260926 15:00:11.678236 8687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:36385 every 8 connection(s)
I20260926 15:00:11.678556 8620 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:00:11.679606 8688 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:11.682003 8688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99: Bootstrap starting.
I20260926 15:00:11.682699 8688 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:11.683023 8688 log.cc:824] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:11.683949 8688 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99: No bootstrap required, opened a new log
I20260926 15:00:11.684969 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 8620
I20260926 15:00:11.685057 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:00:11.685763 8600 external_mini_cluster.cc:1428] Setting key e83877643f00833b184dc7b974caf631
I20260926 15:00:11.685776 8688 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9821273ab0634f009dbb0e6c6d30da99" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36385 } }
I20260926 15:00:11.685992 8688 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:11.686064 8688 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9821273ab0634f009dbb0e6c6d30da99, State: Initialized, Role: FOLLOWER
I20260926 15:00:11.686242 8688 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9821273ab0634f009dbb0e6c6d30da99" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36385 } }
I20260926 15:00:11.686342 8688 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:00:11.686376 8688 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:00:11.686419 8688 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:11.687745 8688 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9821273ab0634f009dbb0e6c6d30da99" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36385 } }
I20260926 15:00:11.687870 8688 leader_election.cc:302] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9821273ab0634f009dbb0e6c6d30da99; no voters:
I20260926 15:00:11.688037 8688 leader_election.cc:288] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:00:11.688091 8693 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:11.688244 8693 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99 [term 1 LEADER]: Becoming Leader. State: Replica: 9821273ab0634f009dbb0e6c6d30da99, State: Running, Role: LEADER
I20260926 15:00:11.688310 8688 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:11.688345 8693 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9821273ab0634f009dbb0e6c6d30da99" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36385 } }
I20260926 15:00:11.688715 8695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9821273ab0634f009dbb0e6c6d30da99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9821273ab0634f009dbb0e6c6d30da99" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36385 } } }
I20260926 15:00:11.688799 8695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:11.688962 8695 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9821273ab0634f009dbb0e6c6d30da99. Latest consensus state: current_term: 1 leader_uuid: "9821273ab0634f009dbb0e6c6d30da99" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9821273ab0634f009dbb0e6c6d30da99" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36385 } } }
I20260926 15:00:11.689033 8695 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:11.689172 8701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:00:11.689941 8701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:00:11.691610 8701 catalog_manager.cc:1347] Generated new cluster ID: a70cf71a638a46f2bb5fa75d2548dfcc
I20260926 15:00:11.691677 8701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 15:00:11 dist-test-slave-jnhs krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:11.702415 8691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:11.686751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58092 (local address 127.8.102.62:36385)
0926 15:00:11.686961 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:11.686970 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:11.687394 (+ 424us) server_negotiation.cc:407] Connection header received
0926 15:00:11.690148 (+ 2754us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:11.690162 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:11.690211 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:11.690352 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:11.692359 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:11.693319 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:11.694406 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:11.694929 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:11.695063 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:11.698463 (+ 3400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:11.698473 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:11.698473 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:11.698485 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:11.700304 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:11.700784 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:11.700787 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:11.700787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:11.700887 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:11.701168 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:11.701169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:11.701170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:11.701425 (+ 255us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:11.701594 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:11.701893 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 15:00:11.702034 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147,"thread_start_us":115,"threads_started":1}
I20260926 15:00:11.722390 8701 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:00:11.723184 8701 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:00:11.730014 8701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9821273ab0634f009dbb0e6c6d30da99: Generated new TSK 0
I20260926 15:00:11.730237 8701 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
I20260926 15:00:11.758318 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36385
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:11.843056 8714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:11.843307 8714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:11.843346 8714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:11.844578 8714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:11.844877 8714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:11.845006 8714 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:00:11.846851 8714 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36385
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8714
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:11.847164 8714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:11.847435 8714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 15:00:11.853056 8714 server_base.cc:1034] running on GCE node
W20260926 15:00:11.853181 8720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:11.853070 8722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:11.853076 8719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:11.853670 8714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:11.853979 8714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:11.855166 8714 hybrid_clock.cc:648] HybridClock initialized: now 1790434811855141 us; error 41 us; skew 500 ppm
Sep 26 15:00:11 dist-test-slave-jnhs krb5kdc[8607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434811, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:11.858014 8714 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:00:11.858476 8714 webserver.cc:460] Webserver started at http://127.8.102.1:34135/ using document root <none> and password file <none>
I20260926 15:00:11.858786 8714 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:11.858855 8714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:11.859022 8714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:11.860119 8714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "b831dbb53742493ead14cff6895b9a65"
format_stamp: "Formatted at 2026-09-26 15:00:11 on dist-test-slave-jnhs"
server_key: "9952db9faae7c021c467f312ce06e628"
server_key_iv: "115f446c785ae4b018da32ad1a9450bf"
server_key_version: "encryptionkey@0"
I20260926 15:00:11.860502 8714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "b831dbb53742493ead14cff6895b9a65"
format_stamp: "Formatted at 2026-09-26 15:00:11 on dist-test-slave-jnhs"
server_key: "9952db9faae7c021c467f312ce06e628"
server_key_iv: "115f446c785ae4b018da32ad1a9450bf"
server_key_version: "encryptionkey@0"
I20260926 15:00:11.861876 8714 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 15:00:11.862874 8729 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:11.863123 8714 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:11.863245 8714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "b831dbb53742493ead14cff6895b9a65"
format_stamp: "Formatted at 2026-09-26 15:00:11 on dist-test-slave-jnhs"
server_key: "9952db9faae7c021c467f312ce06e628"
server_key_iv: "115f446c785ae4b018da32ad1a9450bf"
server_key_version: "encryptionkey@0"
I20260926 15:00:11.863335 8714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:11.920868 8714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:11.923380 8714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:11.923530 8714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:11.923810 8714 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:11.924198 8714 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:11.924255 8714 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:11.924291 8714 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:11.924319 8714 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:11.930179 8714 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:35391
I20260926 15:00:11.930258 8842 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:35391 every 8 connection(s)
I20260926 15:00:11.930614 8714 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:00:11.934816 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 8714
I20260926 15:00:11.934921 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:00:11.935072 8600 external_mini_cluster.cc:1428] Setting key b378f1b580cdea0bee4dd938e42ccc02
Sep 26 15:00:11 dist-test-slave-jnhs krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434811, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:11.941212 8691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:11.931457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:43461 (local address 127.8.102.62:36385)
0926 15:00:11.931581 (+ 124us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:11.931583 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:11.931799 (+ 216us) server_negotiation.cc:407] Connection header received
0926 15:00:11.932332 (+ 533us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:11.932333 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:11.932348 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:11.932422 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:11.933607 (+ 1185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:11.934111 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:11.935487 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:11.935725 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:11.935821 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:11.938208 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:11.938210 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:11.938210 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:11.938217 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:11.939709 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:11.940141 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:11.940143 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:11.940143 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:11.940203 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:11.940510 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:11.940511 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:11.940511 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:11.940635 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:11.940712 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:11.941018 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 15:00:11.941110 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:00:11.941414 8845 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:11.931438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36385 (local address 127.8.102.1:43461)
0926 15:00:11.931646 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:11.931694 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:11.932218 (+ 524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:11.932483 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:11.932487 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:11.933194 (+ 707us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:11.933500 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:11.933508 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:11.934215 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:11.934216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:11.935388 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:11.935391 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:11.935834 (+ 443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:11.935835 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:11.935968 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:11.936572 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:11.936581 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:11.938079 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:11.939802 (+ 1723us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:11.939809 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:11.939812 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:11.940059 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:11.940273 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:11.940274 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:11.940274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:11.940393 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:11.940704 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:11.940710 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:11.940868 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 15:00:11.940985 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 15:00:11.941165 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":121,"threads_started":1}
I20260926 15:00:11.942022 8843 heartbeater.cc:347] Connected to a master server at 127.8.102.62:36385
I20260926 15:00:11.942133 8843 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:11.942322 8843 heartbeater.cc:511] Master 127.8.102.62:36385 requested a full tablet report, sending...
I20260926 15:00:11.943029 8652 ts_manager.cc:194] Registered new tserver with Master: b831dbb53742493ead14cff6895b9a65 (127.8.102.1:35391)
WARNING: no policy specified for kudu/127.8.102.2@KRBTEST.COM; defaulting to no policy
I20260926 15:00:11.943989 8652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:43461
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
I20260926 15:00:11.961431 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36385
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:12.043654 8848 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:12.043874 8848 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:12.043900 8848 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:12.045133 8848 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:12.045181 8848 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:12.045210 8848 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:00:12.046734 8848 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36385
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8848
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:12.046990 8848 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:12.047246 8848 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:12.052525 8856 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:12.052559 8853 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:12.052542 8848 server_base.cc:1034] running on GCE node
W20260926 15:00:12.052510 8854 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:12.052950 8848 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:12.053196 8848 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:12.054333 8848 hybrid_clock.cc:648] HybridClock initialized: now 1790434812054321 us; error 31 us; skew 500 ppm
Sep 26 15:00:12 dist-test-slave-jnhs krb5kdc[8607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434812, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:12.057148 8848 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:00:12.057683 8848 webserver.cc:460] Webserver started at http://127.8.102.2:40339/ using document root <none> and password file <none>
I20260926 15:00:12.057919 8848 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:12.057965 8848 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:12.058111 8848 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:12.059242 8848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "89a7ba505def4921b140090f39ebff35"
format_stamp: "Formatted at 2026-09-26 15:00:12 on dist-test-slave-jnhs"
server_key: "891577c5f25be67735bbd47a79add3d7"
server_key_iv: "3d22b487cb554defc6474cd3c1775bc3"
server_key_version: "encryptionkey@0"
I20260926 15:00:12.059621 8848 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "89a7ba505def4921b140090f39ebff35"
format_stamp: "Formatted at 2026-09-26 15:00:12 on dist-test-slave-jnhs"
server_key: "891577c5f25be67735bbd47a79add3d7"
server_key_iv: "3d22b487cb554defc6474cd3c1775bc3"
server_key_version: "encryptionkey@0"
I20260926 15:00:12.060997 8848 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:00:12.061743 8863 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:12.061930 8848 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 15:00:12.062022 8848 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "89a7ba505def4921b140090f39ebff35"
format_stamp: "Formatted at 2026-09-26 15:00:12 on dist-test-slave-jnhs"
server_key: "891577c5f25be67735bbd47a79add3d7"
server_key_iv: "3d22b487cb554defc6474cd3c1775bc3"
server_key_version: "encryptionkey@0"
I20260926 15:00:12.062096 8848 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:12.072966 8848 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:12.075539 8848 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:12.075690 8848 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:12.075964 8848 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:12.076308 8848 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:12.076345 8848 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:12.076367 8848 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:12.076380 8848 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:12.082613 8848 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:33747
I20260926 15:00:12.082716 8976 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:33747 every 8 connection(s)
I20260926 15:00:12.083076 8848 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:00:12.086539 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 8848
I20260926 15:00:12.086694 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:00:12.086887 8600 external_mini_cluster.cc:1428] Setting key a33f5defd871cc5d1f91fe505387f9fd
Sep 26 15:00:12 dist-test-slave-jnhs krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434812, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:12.093782 8691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:12.083789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:43379 (local address 127.8.102.62:36385)
0926 15:00:12.083838 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:12.083840 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:12.084069 (+ 229us) server_negotiation.cc:407] Connection header received
0926 15:00:12.084557 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:12.084558 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:12.084569 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:12.084629 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:12.085596 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:12.086103 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:12.087506 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:12.087789 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:12.087848 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:12.090591 (+ 2743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:12.090592 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:12.090593 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:12.090601 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:12.092167 (+ 1566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:12.092742 (+ 575us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:12.092742 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:12.092743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:12.092800 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:12.093108 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:12.093109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:12.093109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:12.093219 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:12.093270 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:12.093635 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 15:00:12.093715 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 15:00:12.094063 8979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:12.083762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36385 (local address 127.8.102.2:43379)
0926 15:00:12.083926 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:12.083968 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:12.084455 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:12.084685 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:12.084688 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:12.085274 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:12.085472 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:12.085477 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:12.086218 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:12.086219 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:12.087399 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:12.087401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:12.087877 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:12.087878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:12.088051 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:12.088605 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:12.088616 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:12.090487 (+ 1871us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:12.092262 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:12.092271 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:12.092274 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:12.092647 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:12.092873 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:12.092874 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:12.092874 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:12.093037 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:12.093285 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:12.093291 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:12.093486 (+ 195us) client_negotiation.cc:769] Sending connection context
0926 15:00:12.093617 (+ 131us) client_negotiation.cc:240] Negotiation successful
0926 15:00:12.093804 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260926 15:00:12.094759 8977 heartbeater.cc:347] Connected to a master server at 127.8.102.62:36385
I20260926 15:00:12.094852 8977 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:12.095070 8977 heartbeater.cc:511] Master 127.8.102.62:36385 requested a full tablet report, sending...
I20260926 15:00:12.095517 8652 ts_manager.cc:194] Registered new tserver with Master: 89a7ba505def4921b140090f39ebff35 (127.8.102.2:33747)
I20260926 15:00:12.095898 8652 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:43379
WARNING: no policy specified for kudu/127.8.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
I20260926 15:00:12.114271 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36385
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:37353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:12.197965 8982 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:12.198189 8982 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:12.198215 8982 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:12.199501 8982 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:12.199558 8982 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:12.199586 8982 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:00:12.201016 8982 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:37353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36385
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.8982
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:12.201212 8982 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:12.201396 8982 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:12.207206 8990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:12.207207 8987 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:12.207883 8988 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:12.208038 8982 server_base.cc:1034] running on GCE node
I20260926 15:00:12.208246 8982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:12.208467 8982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:12.209622 8982 hybrid_clock.cc:648] HybridClock initialized: now 1790434812209603 us; error 33 us; skew 500 ppm
Sep 26 15:00:12 dist-test-slave-jnhs krb5kdc[8607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434812, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:12.212338 8982 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:00:12.212863 8982 webserver.cc:460] Webserver started at http://127.8.102.3:38515/ using document root <none> and password file <none>
I20260926 15:00:12.213168 8982 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:12.213241 8982 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:12.213405 8982 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:12.214509 8982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "a4c15f51066e48f584586dc5018703da"
format_stamp: "Formatted at 2026-09-26 15:00:12 on dist-test-slave-jnhs"
server_key: "1b5c12b05d16ac519b8023f5fc23b49e"
server_key_iv: "8092f4d52e1aaf67eee7aced2fd6ac5f"
server_key_version: "encryptionkey@0"
I20260926 15:00:12.214964 8982 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "a4c15f51066e48f584586dc5018703da"
format_stamp: "Formatted at 2026-09-26 15:00:12 on dist-test-slave-jnhs"
server_key: "1b5c12b05d16ac519b8023f5fc23b49e"
server_key_iv: "8092f4d52e1aaf67eee7aced2fd6ac5f"
server_key_version: "encryptionkey@0"
I20260926 15:00:12.216390 8982 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:12.217232 8997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:12.217433 8982 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:12.217533 8982 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "a4c15f51066e48f584586dc5018703da"
format_stamp: "Formatted at 2026-09-26 15:00:12 on dist-test-slave-jnhs"
server_key: "1b5c12b05d16ac519b8023f5fc23b49e"
server_key_iv: "8092f4d52e1aaf67eee7aced2fd6ac5f"
server_key_version: "encryptionkey@0"
I20260926 15:00:12.217623 8982 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:12.261422 8982 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:12.263809 8982 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:12.264093 8982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:12.264353 8982 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:12.264715 8982 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:12.264772 8982 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:12.264814 8982 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:12.264863 8982 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:12.270980 8982 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:36267
I20260926 15:00:12.271041 9110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:36267 every 8 connection(s)
I20260926 15:00:12.271356 8982 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
Sep 26 15:00:12 dist-test-slave-jnhs krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434812, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:12.280108 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 8982
I20260926 15:00:12.280196 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:00:12.280405 8600 external_mini_cluster.cc:1428] Setting key 3176389a773c867bb1aa09dfd6099eb4
I20260926 15:00:12.280983 8691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:12.272111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:60069 (local address 127.8.102.62:36385)
0926 15:00:12.272199 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:12.272202 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:12.272364 (+ 162us) server_negotiation.cc:407] Connection header received
0926 15:00:12.272836 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:12.272837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:12.272847 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:12.272908 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:12.273880 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:12.274368 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:12.275496 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:12.275690 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:12.275748 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:12.277880 (+ 2132us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:12.277882 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:12.277882 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:12.277888 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:12.279463 (+ 1575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:12.279834 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:12.279835 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:12.279835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:12.279884 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:12.280190 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:12.280191 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:12.280192 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:12.280328 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:12.280383 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:12.280778 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 15:00:12.280883 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:00:12.281172 9113 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:12.272059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36385 (local address 127.8.102.3:60069)
0926 15:00:12.272229 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:12.272273 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:12.272752 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:12.272990 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:12.272993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:12.273582 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:12.273781 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:12.273786 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:12.274451 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:12.274452 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:12.275404 (+ 952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:12.275406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:12.275772 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:12.275773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:12.275878 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:12.276391 (+ 513us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:12.276410 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:12.277788 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:12.279542 (+ 1754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:12.279549 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:12.279550 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:12.279768 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:12.279956 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:12.279958 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:12.279958 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:12.280100 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:12.280394 (+ 294us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:12.280400 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:12.280612 (+ 212us) client_negotiation.cc:769] Sending connection context
0926 15:00:12.280729 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 15:00:12.280938 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":92,"threads_started":1}
I20260926 15:00:12.281661 9111 heartbeater.cc:347] Connected to a master server at 127.8.102.62:36385
I20260926 15:00:12.281773 9111 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:12.281972 9111 heartbeater.cc:511] Master 127.8.102.62:36385 requested a full tablet report, sending...
I20260926 15:00:12.282383 8651 ts_manager.cc:194] Registered new tserver with Master: a4c15f51066e48f584586dc5018703da (127.8.102.3:36267)
I20260926 15:00:12.282871 8651 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:60069
I20260926 15:00:12.292930 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 15:00:12 dist-test-slave-jnhs krb5kdc[8607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434812, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 15:00:12 dist-test-slave-jnhs krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434812, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.102.1@KRBTEST.COM
I20260926 15:00:12.317394 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:12.308977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.8.102.1:35391)
0926 15:00:12.309154 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:12.309160 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:12.309174 (+ 14us) server_negotiation.cc:407] Connection header received
0926 15:00:12.309205 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:12.309214 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:12.309230 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:12.309312 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:12.309911 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:12.310726 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:12.311486 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:12.311830 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:12.311926 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:12.314329 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:12.314337 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:12.314337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:12.314345 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:12.316090 (+ 1745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:12.316503 (+ 413us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:12.316507 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:12.316508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:12.316589 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:12.316854 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:12.316855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:12.316855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:12.317043 (+ 188us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:12.317174 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:12.317237 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 15:00:12.317309 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":77,"threads_started":1}
W20260926 15:00:12.317950 8757 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54992
Sep 26 15:00:12 dist-test-slave-jnhs krb5kdc[8607](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 15:00:12 dist-test-slave-jnhs krb5kdc[8607](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.1@KRBTEST.COM
I20260926 15:00:12.341231 9118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:12.333895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54994 (local address 127.8.102.1:35391)
0926 15:00:12.333978 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:12.333980 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:12.334088 (+ 108us) server_negotiation.cc:407] Connection header received
0926 15:00:12.334117 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:12.334119 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:12.334133 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:12.334178 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:12.334818 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:12.335301 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:12.336061 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:12.336259 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:12.336317 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:12.338701 (+ 2384us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:12.338702 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:12.338703 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:12.338713 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:12.339959 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:12.340323 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:12.340324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:12.340324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:12.340384 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:12.340662 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:12.340663 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:12.340664 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:12.340783 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:12.340895 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:12.341006 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 15:00:12.341089 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:00:12.341998 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 8714
I20260926 15:00:12.347152 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 8848
I20260926 15:00:12.351840 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 8982
I20260926 15:00:12.356698 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 8620
2026-09-26T15:00:12Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (2001 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:00:12 dist-test-slave-jnhs krb5kdc[9125](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:00:12 dist-test-slave-jnhs krb5kdc[9125](info): set up 2 sockets
Sep 26 15:00:12 dist-test-slave-jnhs krb5kdc[9125](info): commencing operation
krb5kdc: starting...
W20260926 15:00:13.402275 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:00:13 dist-test-slave-jnhs krb5kdc[9125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:00:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:00:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:00:13.525739 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:44219
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:44219
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:13.606769 9138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:13.607056 9138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:13.607097 9138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:13.608395 9138 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:13.608490 9138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:13.608520 9138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:13.608543 9138 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:13.608570 9138 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:00:13.610173 9138 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:44219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:44219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:13.610500 9138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:13.610767 9138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:13.615979 9144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:13.615979 9146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:13.615998 9138 server_base.cc:1034] running on GCE node
W20260926 15:00:13.615984 9143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:13.616487 9138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:13.616822 9138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:13.617974 9138 hybrid_clock.cc:648] HybridClock initialized: now 1790434813617956 us; error 48 us; skew 500 ppm
Sep 26 15:00:13 dist-test-slave-jnhs krb5kdc[9125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434813, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:13.620864 9138 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:00:13.621374 9138 webserver.cc:460] Webserver started at http://127.8.102.62:43407/ using document root <none> and password file <none>
I20260926 15:00:13.621632 9138 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:13.621685 9138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:13.621878 9138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:13.623178 9138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "1fa82e6a2fce43b6b6ea8d51706acd01"
format_stamp: "Formatted at 2026-09-26 15:00:13 on dist-test-slave-jnhs"
server_key: "c47d501a03e176868742c05c153d008e"
server_key_iv: "0a28d0c2091e1667ef61465c7a68d1b5"
server_key_version: "encryptionkey@0"
I20260926 15:00:13.623600 9138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "1fa82e6a2fce43b6b6ea8d51706acd01"
format_stamp: "Formatted at 2026-09-26 15:00:13 on dist-test-slave-jnhs"
server_key: "c47d501a03e176868742c05c153d008e"
server_key_iv: "0a28d0c2091e1667ef61465c7a68d1b5"
server_key_version: "encryptionkey@0"
I20260926 15:00:13.625118 9138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:00:13.625986 9153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:13.626192 9138 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:13.626303 9138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "1fa82e6a2fce43b6b6ea8d51706acd01"
format_stamp: "Formatted at 2026-09-26 15:00:13 on dist-test-slave-jnhs"
server_key: "c47d501a03e176868742c05c153d008e"
server_key_iv: "0a28d0c2091e1667ef61465c7a68d1b5"
server_key_version: "encryptionkey@0"
I20260926 15:00:13.626392 9138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:13.648339 9138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:13.650553 9138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:13.650803 9138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:13.654752 9138 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:44219
I20260926 15:00:13.654814 9205 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:44219 every 8 connection(s)
I20260926 15:00:13.655200 9138 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:00:13.656090 9206 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:13.658461 9206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01: Bootstrap starting.
I20260926 15:00:13.659107 9206 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:13.659413 9206 log.cc:824] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:13.660166 9206 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01: No bootstrap required, opened a new log
I20260926 15:00:13.660782 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 9138
I20260926 15:00:13.660882 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:00:13.661037 8600 external_mini_cluster.cc:1428] Setting key ee577a3029cb5cacad68ea763f172aa4
I20260926 15:00:13.662397 9206 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa82e6a2fce43b6b6ea8d51706acd01" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44219 } }
I20260926 15:00:13.662581 9206 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:13.662622 9206 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa82e6a2fce43b6b6ea8d51706acd01, State: Initialized, Role: FOLLOWER
I20260926 15:00:13.662809 9206 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa82e6a2fce43b6b6ea8d51706acd01" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44219 } }
I20260926 15:00:13.662904 9206 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:00:13.662946 9206 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:00:13.663007 9206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:13.664037 9206 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa82e6a2fce43b6b6ea8d51706acd01" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44219 } }
I20260926 15:00:13.664134 9206 leader_election.cc:302] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1fa82e6a2fce43b6b6ea8d51706acd01; no voters:
I20260926 15:00:13.664366 9206 leader_election.cc:288] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:00:13.664388 9211 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:13.664505 9211 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01 [term 1 LEADER]: Becoming Leader. State: Replica: 1fa82e6a2fce43b6b6ea8d51706acd01, State: Running, Role: LEADER
I20260926 15:00:13.664590 9211 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa82e6a2fce43b6b6ea8d51706acd01" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44219 } }
I20260926 15:00:13.664660 9206 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:13.664901 9211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1fa82e6a2fce43b6b6ea8d51706acd01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa82e6a2fce43b6b6ea8d51706acd01" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44219 } } }
I20260926 15:00:13.664973 9211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:13.665102 9211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1fa82e6a2fce43b6b6ea8d51706acd01. Latest consensus state: current_term: 1 leader_uuid: "1fa82e6a2fce43b6b6ea8d51706acd01" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1fa82e6a2fce43b6b6ea8d51706acd01" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44219 } } }
I20260926 15:00:13.665187 9211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:13.665400 9215 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:00:13.665938 9215 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 15:00:13 dist-test-slave-jnhs krb5kdc[9125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:13.671265 9215 catalog_manager.cc:1347] Generated new cluster ID: 0113aeb585104c409c019839035c5be2
I20260926 15:00:13.671360 9215 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:00:13.673107 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:13.661590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49662 (local address 127.8.102.62:44219)
0926 15:00:13.661772 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:13.661780 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:13.661805 (+ 25us) server_negotiation.cc:407] Connection header received
0926 15:00:13.662370 (+ 565us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:13.662376 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:13.662426 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:13.662543 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:13.663378 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:13.664473 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:13.665567 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:13.666283 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:13.666421 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:13.669287 (+ 2866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:13.669291 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:13.669292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:13.669304 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:13.671269 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:13.671808 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:13.671811 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:13.671812 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:13.671964 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:13.672215 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:13.672216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:13.672217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:13.672489 (+ 272us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:13.672659 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:13.672771 (+ 112us) server_negotiation.cc:299] Negotiation successful
0926 15:00:13.672928 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":95,"threads_started":1}
I20260926 15:00:13.676347 9215 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:00:13.676996 9215 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:00:13.688066 9215 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1fa82e6a2fce43b6b6ea8d51706acd01: Generated new TSK 0
I20260926 15:00:13.688287 9215 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
I20260926 15:00:13.723026 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44219
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:13.810889 9232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:13.811105 9232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:13.811132 9232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:13.812412 9232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:13.812457 9232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:13.812489 9232 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:00:13.814083 9232 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44219
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9232
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:13.814353 9232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:13.814603 9232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:13.819830 9240 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:13.819862 9237 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:13.819840 9238 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:13.820220 9232 server_base.cc:1034] running on GCE node
I20260926 15:00:13.820457 9232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:13.820740 9232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:13.821918 9232 hybrid_clock.cc:648] HybridClock initialized: now 1790434813821894 us; error 40 us; skew 500 ppm
Sep 26 15:00:13 dist-test-slave-jnhs krb5kdc[9125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434813, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:13.824982 9232 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:00:13.825524 9232 webserver.cc:460] Webserver started at http://127.8.102.1:34419/ using document root <none> and password file <none>
I20260926 15:00:13.825737 9232 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:13.825783 9232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:13.825888 9232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:13.827138 9232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "74b5279690b643069d4dd544d2d32633"
format_stamp: "Formatted at 2026-09-26 15:00:13 on dist-test-slave-jnhs"
server_key: "43b5be564315198bf2ffce8b8a672f10"
server_key_iv: "22ce2235561a3ac44dff7f8ce7f266d0"
server_key_version: "encryptionkey@0"
I20260926 15:00:13.827490 9232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "74b5279690b643069d4dd544d2d32633"
format_stamp: "Formatted at 2026-09-26 15:00:13 on dist-test-slave-jnhs"
server_key: "43b5be564315198bf2ffce8b8a672f10"
server_key_iv: "22ce2235561a3ac44dff7f8ce7f266d0"
server_key_version: "encryptionkey@0"
I20260926 15:00:13.829026 9232 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:13.829993 9247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:13.830250 9232 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:13.830375 9232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "74b5279690b643069d4dd544d2d32633"
format_stamp: "Formatted at 2026-09-26 15:00:13 on dist-test-slave-jnhs"
server_key: "43b5be564315198bf2ffce8b8a672f10"
server_key_iv: "22ce2235561a3ac44dff7f8ce7f266d0"
server_key_version: "encryptionkey@0"
I20260926 15:00:13.830461 9232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:13.839841 9232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:13.841835 9232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:13.842010 9232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:13.842252 9232 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:13.842595 9232 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:13.842720 9232 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:13.842774 9232 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:13.842801 9232 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:13.848779 9232 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:46049
I20260926 15:00:13.848826 9360 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:46049 every 8 connection(s)
I20260926 15:00:13.849157 9232 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
Sep 26 15:00:13 dist-test-slave-jnhs krb5kdc[9125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434813, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:13.858556 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 9232
I20260926 15:00:13.858696 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:00:13.858866 8600 external_mini_cluster.cc:1428] Setting key 699f947c693f33a1d8d5e4a1a04d053a
I20260926 15:00:13.859632 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:13.849914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:56577 (local address 127.8.102.62:44219)
0926 15:00:13.849959 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:13.849961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:13.850246 (+ 285us) server_negotiation.cc:407] Connection header received
0926 15:00:13.851335 (+ 1089us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:13.851337 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:13.851354 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:13.851411 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:13.852092 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:13.852601 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:13.853678 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:13.853891 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:13.853976 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:13.856359 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:13.856361 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:13.856361 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:13.856380 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:13.857918 (+ 1538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:13.858446 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:13.858447 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:13.858447 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:13.858512 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:13.858846 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:13.858847 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:13.858848 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:13.859026 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:13.859125 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:13.859434 (+ 309us) server_negotiation.cc:299] Negotiation successful
0926 15:00:13.859540 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 15:00:13.859838 9363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:13.849921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44219 (local address 127.8.102.1:56577)
0926 15:00:13.850110 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:13.850146 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:13.851203 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:13.851504 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:13.851508 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:13.851775 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:13.851970 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:13.851975 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:13.852704 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:13.852705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:13.853572 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:13.853575 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:13.853984 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:13.853985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:13.854113 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:13.854624 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:13.854676 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:13.856232 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:13.858077 (+ 1845us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:13.858084 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:13.858087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:13.858335 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:13.858582 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:13.858583 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:13.858583 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:13.858762 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:13.859107 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:13.859114 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:13.859277 (+ 163us) client_negotiation.cc:769] Sending connection context
0926 15:00:13.859387 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 15:00:13.859562 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":86,"threads_started":1}
I20260926 15:00:13.860405 9361 heartbeater.cc:347] Connected to a master server at 127.8.102.62:44219
I20260926 15:00:13.860536 9361 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:13.860774 9361 heartbeater.cc:511] Master 127.8.102.62:44219 requested a full tablet report, sending...
I20260926 15:00:13.861325 9170 ts_manager.cc:194] Registered new tserver with Master: 74b5279690b643069d4dd544d2d32633 (127.8.102.1:46049)
I20260926 15:00:13.861969 9170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:56577
WARNING: no policy specified for kudu/127.8.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
I20260926 15:00:13.886111 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44219
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:13.974772 9366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:13.974984 9366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:13.975010 9366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:13.976249 9366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:13.976300 9366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:13.976327 9366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:00:13.977921 9366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44219
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9366
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:13.978183 9366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:13.978442 9366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:13.983631 9372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:13.983628 9374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:13.983626 9371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:13.983704 9366 server_base.cc:1034] running on GCE node
I20260926 15:00:13.984181 9366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:13.984472 9366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:13.985702 9366 hybrid_clock.cc:648] HybridClock initialized: now 1790434813985685 us; error 28 us; skew 500 ppm
Sep 26 15:00:13 dist-test-slave-jnhs krb5kdc[9125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434813, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:13.988448 9366 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:00:13.988946 9366 webserver.cc:460] Webserver started at http://127.8.102.2:43977/ using document root <none> and password file <none>
I20260926 15:00:13.989187 9366 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:13.989256 9366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:13.989414 9366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:13.990514 9366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "f2daa0b1b1004aeb9cc0d5d886d9512f"
format_stamp: "Formatted at 2026-09-26 15:00:13 on dist-test-slave-jnhs"
server_key: "eb6b83d721307d3a710b64e43926adcc"
server_key_iv: "f0a80fb4a8c74f38df8feecfd591847f"
server_key_version: "encryptionkey@0"
I20260926 15:00:13.990979 9366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "f2daa0b1b1004aeb9cc0d5d886d9512f"
format_stamp: "Formatted at 2026-09-26 15:00:13 on dist-test-slave-jnhs"
server_key: "eb6b83d721307d3a710b64e43926adcc"
server_key_iv: "f0a80fb4a8c74f38df8feecfd591847f"
server_key_version: "encryptionkey@0"
I20260926 15:00:13.992399 9366 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:13.993286 9381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:13.993647 9366 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:13.993788 9366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "f2daa0b1b1004aeb9cc0d5d886d9512f"
format_stamp: "Formatted at 2026-09-26 15:00:13 on dist-test-slave-jnhs"
server_key: "eb6b83d721307d3a710b64e43926adcc"
server_key_iv: "f0a80fb4a8c74f38df8feecfd591847f"
server_key_version: "encryptionkey@0"
I20260926 15:00:13.993891 9366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:14.003412 9366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:14.005502 9366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:14.005682 9366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:14.005954 9366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:14.006304 9366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:14.006363 9366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:14.006407 9366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:14.006438 9366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:14.012328 9366 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:41469
I20260926 15:00:14.012379 9494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:41469 every 8 connection(s)
I20260926 15:00:14.012714 9366 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
Sep 26 15:00:14 dist-test-slave-jnhs krb5kdc[9125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434813, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:14.021541 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 9366
I20260926 15:00:14.021662 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:00:14.021822 8600 external_mini_cluster.cc:1428] Setting key c141a9fd0b1a57105b214ece130c87e6
I20260926 15:00:14.023119 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:14.013484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:35987 (local address 127.8.102.62:44219)
0926 15:00:14.013559 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:14.013562 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:14.013807 (+ 245us) server_negotiation.cc:407] Connection header received
0926 15:00:14.014942 (+ 1135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:14.014944 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:14.014961 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:14.015030 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:14.015760 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.016306 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.017379 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.017585 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.017647 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:14.020056 (+ 2409us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:14.020057 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:14.020057 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:14.020064 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:14.021369 (+ 1305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:14.021899 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:14.021900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:14.021901 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:14.021975 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:14.022351 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:14.022352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:14.022352 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:14.022497 (+ 145us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:14.022585 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:14.022963 (+ 378us) server_negotiation.cc:299] Negotiation successful
0926 15:00:14.023054 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 15:00:14.023458 9497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:14.013474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44219 (local address 127.8.102.2:35987)
0926 15:00:14.013679 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:14.013718 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:14.014775 (+ 1057us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:14.015115 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:14.015119 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:14.015414 (+ 295us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:14.015623 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:14.015627 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.016414 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.016414 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:14.017282 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:14.017285 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.017686 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.017686 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:14.017809 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:14.018412 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:14.018421 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:14.019957 (+ 1536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:14.021503 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:14.021510 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:14.021513 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:14.021820 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:14.022072 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:14.022083 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:14.022083 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:14.022280 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:14.022583 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:14.022588 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:14.022802 (+ 214us) client_negotiation.cc:769] Sending connection context
0926 15:00:14.022937 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 15:00:14.023132 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":87,"threads_started":1}
I20260926 15:00:14.024027 9495 heartbeater.cc:347] Connected to a master server at 127.8.102.62:44219
I20260926 15:00:14.024161 9495 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:14.024387 9495 heartbeater.cc:511] Master 127.8.102.62:44219 requested a full tablet report, sending...
I20260926 15:00:14.024833 9170 ts_manager.cc:194] Registered new tserver with Master: f2daa0b1b1004aeb9cc0d5d886d9512f (127.8.102.2:41469)
I20260926 15:00:14.025388 9170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:35987
WARNING: no policy specified for kudu/127.8.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
I20260926 15:00:14.050890 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44219
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:14.133723 9500 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:14.133996 9500 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:14.134024 9500 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:14.135399 9500 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:14.135457 9500 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:14.135488 9500 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:00:14.137156 9500 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44219
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9500
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:14.137445 9500 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:14.137722 9500 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:14.143072 9506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:14.143083 9508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:14.143163 9505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:14.143440 9500 server_base.cc:1034] running on GCE node
I20260926 15:00:14.143692 9500 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:14.143953 9500 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:14.145128 9500 hybrid_clock.cc:648] HybridClock initialized: now 1790434814145105 us; error 44 us; skew 500 ppm
Sep 26 15:00:14 dist-test-slave-jnhs krb5kdc[9125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434814, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:14.147943 9500 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:00:14.148409 9500 webserver.cc:460] Webserver started at http://127.8.102.3:41683/ using document root <none> and password file <none>
I20260926 15:00:14.148649 9500 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:14.148718 9500 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:14.148875 9500 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:14.149942 9500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "c87accce8a18422aa2e7ac8e762869ba"
format_stamp: "Formatted at 2026-09-26 15:00:14 on dist-test-slave-jnhs"
server_key: "d0e87b240c4d2ba8873e6f618dbabef9"
server_key_iv: "c4a7aae16690ecd35a2a824bf4219623"
server_key_version: "encryptionkey@0"
I20260926 15:00:14.150310 9500 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "c87accce8a18422aa2e7ac8e762869ba"
format_stamp: "Formatted at 2026-09-26 15:00:14 on dist-test-slave-jnhs"
server_key: "d0e87b240c4d2ba8873e6f618dbabef9"
server_key_iv: "c4a7aae16690ecd35a2a824bf4219623"
server_key_version: "encryptionkey@0"
I20260926 15:00:14.151909 9500 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 15:00:14.152767 9515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:14.152957 9500 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:14.153077 9500 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "c87accce8a18422aa2e7ac8e762869ba"
format_stamp: "Formatted at 2026-09-26 15:00:14 on dist-test-slave-jnhs"
server_key: "d0e87b240c4d2ba8873e6f618dbabef9"
server_key_iv: "c4a7aae16690ecd35a2a824bf4219623"
server_key_version: "encryptionkey@0"
I20260926 15:00:14.153158 9500 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:14.174799 9500 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:14.177214 9500 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:14.177385 9500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:14.177630 9500 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:14.177953 9500 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:14.178009 9500 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:14.178052 9500 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:14.178130 9500 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:14.184381 9500 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:36819
I20260926 15:00:14.184425 9628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:36819 every 8 connection(s)
I20260926 15:00:14.184801 9500 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:00:14.186179 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 9500
I20260926 15:00:14.186291 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:00:14.186436 8600 external_mini_cluster.cc:1428] Setting key fac2510e26670182ad14454ba79094d3
Sep 26 15:00:14 dist-test-slave-jnhs krb5kdc[9125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434814, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:14.195411 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:14.185612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:36905 (local address 127.8.102.62:44219)
0926 15:00:14.185666 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:14.185669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:14.185914 (+ 245us) server_negotiation.cc:407] Connection header received
0926 15:00:14.187268 (+ 1354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:14.187269 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:14.187281 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:14.187345 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:14.188060 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.188735 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.189759 (+ 1024us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.189974 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.190039 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:14.192585 (+ 2546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:14.192586 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:14.192586 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:14.192594 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:14.193996 (+ 1402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:14.194383 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:14.194385 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:14.194385 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:14.194439 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:14.194781 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:14.194782 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:14.194783 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:14.194898 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:14.194952 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:14.195272 (+ 320us) server_negotiation.cc:299] Negotiation successful
0926 15:00:14.195345 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 15:00:14.195540 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:14.185605 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44219 (local address 127.8.102.3:36905)
0926 15:00:14.185785 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:14.185824 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:14.187171 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:14.187446 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:14.187450 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:14.187746 (+ 296us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:14.187949 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:14.187953 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.188819 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.188820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:14.189660 (+ 840us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:14.189662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.190053 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.190054 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:14.190155 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:14.190882 (+ 727us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:14.190892 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:14.192492 (+ 1600us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:14.194079 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:14.194087 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:14.194090 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:14.194310 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:14.194524 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:14.194525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:14.194526 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:14.194724 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:14.194965 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:14.194970 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:14.195100 (+ 130us) client_negotiation.cc:769] Sending connection context
0926 15:00:14.195190 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 15:00:14.195328 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20260926 15:00:14.196135 9629 heartbeater.cc:347] Connected to a master server at 127.8.102.62:44219
I20260926 15:00:14.196252 9629 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:14.196470 9629 heartbeater.cc:511] Master 127.8.102.62:44219 requested a full tablet report, sending...
I20260926 15:00:14.196981 9170 ts_manager.cc:194] Registered new tserver with Master: c87accce8a18422aa2e7ac8e762869ba (127.8.102.3:36819)
I20260926 15:00:14.197513 9170 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:36905
I20260926 15:00:14.198776 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:00:14.206573 9210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:14.200427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49678 (local address 127.8.102.62:44219)
0926 15:00:14.200487 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:14.200489 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:14.200527 (+ 38us) server_negotiation.cc:407] Connection header received
0926 15:00:14.200618 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:14.200619 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:14.200628 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:14.200686 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:14.201297 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.201822 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.202557 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.202830 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.202915 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:14.204109 (+ 1194us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:14.204110 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:14.204111 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:14.204118 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:14.205344 (+ 1226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:14.205724 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:14.205726 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:14.205726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:14.205784 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:14.205991 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:14.205992 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:14.205992 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:14.206156 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:14.206270 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:14.206378 (+ 108us) server_negotiation.cc:299] Negotiation successful
0926 15:00:14.206459 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 15:00:14.208971 9170 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49678:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:00:14.210501 9170 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:00:14.218675 9643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:14.213299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:46049 (local address 127.0.0.1:43788)
0926 15:00:14.213754 (+ 455us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:14.213832 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:14.213929 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:14.214342 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:14.214345 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:14.214347 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:14.214423 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:14.214425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.215736 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.215738 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:14.217276 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:14.217279 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.218308 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.218309 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:14.218404 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:14.218417 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 15:00:14.218487 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 15:00:14.218550 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":385,"thread_start_us":71,"threads_started":1}
I20260926 15:00:14.218918 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:14.213430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43788 (local address 127.8.102.1:46049)
0926 15:00:14.214078 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:14.214085 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:14.214096 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:00:14.214145 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:14.214147 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:14.214163 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:14.214262 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:14.214585 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.215613 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.217390 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.218141 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.218214 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:14.218273 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:14.218795 (+ 522us) server_negotiation.cc:299] Negotiation successful
0926 15:00:14.218839 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":160,"threads_started":1}
I20260926 15:00:14.219920 9641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:14.213334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:41469 (local address 127.0.0.1:33454)
0926 15:00:14.213873 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:14.213882 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:14.213929 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:14.216042 (+ 2113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:14.216043 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:14.216045 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:14.216091 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:14.216092 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.217400 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.217400 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:14.218446 (+ 1046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:14.218448 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.219669 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.219670 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:14.219737 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:14.219743 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 15:00:14.219803 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 15:00:14.219846 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":82,"threads_started":1}
I20260926 15:00:14.219929 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:14.213678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33454 (local address 127.8.102.2:41469)
0926 15:00:14.215812 (+ 2134us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:14.215817 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:14.215826 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:14.215863 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:14.215864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:14.215881 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:14.215956 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:14.216182 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.217283 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.218555 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.219567 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.219666 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:14.219724 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:14.219817 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 15:00:14.219854 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2092,"thread_start_us":82,"threads_started":1}
I20260926 15:00:14.220618 9429 tablet_service.cc:1467] Processing CreateTablet for tablet 1f377cd331c24e148fbcbc7085870572 (DEFAULT_TABLE table=test-table [id=63bdfc04118340919d01095053dbbc9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:14.220914 9429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f377cd331c24e148fbcbc7085870572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:14.224107 9646 tablet_bootstrap.cc:492] T 1f377cd331c24e148fbcbc7085870572 P f2daa0b1b1004aeb9cc0d5d886d9512f: Bootstrap starting.
I20260926 15:00:14.224858 9646 tablet_bootstrap.cc:654] T 1f377cd331c24e148fbcbc7085870572 P f2daa0b1b1004aeb9cc0d5d886d9512f: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:14.225172 9295 tablet_service.cc:1467] Processing CreateTablet for tablet 1f377cd331c24e148fbcbc7085870572 (DEFAULT_TABLE table=test-table [id=63bdfc04118340919d01095053dbbc9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:14.225430 9646 log.cc:824] T 1f377cd331c24e148fbcbc7085870572 P f2daa0b1b1004aeb9cc0d5d886d9512f: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:14.225553 9295 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f377cd331c24e148fbcbc7085870572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:14.226614 9646 tablet_bootstrap.cc:492] T 1f377cd331c24e148fbcbc7085870572 P f2daa0b1b1004aeb9cc0d5d886d9512f: No bootstrap required, opened a new log
I20260926 15:00:14.226720 9646 ts_tablet_manager.cc:1397] T 1f377cd331c24e148fbcbc7085870572 P f2daa0b1b1004aeb9cc0d5d886d9512f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 15:00:14.228822 9646 raft_consensus.cc:348] T 1f377cd331c24e148fbcbc7085870572 P f2daa0b1b1004aeb9cc0d5d886d9512f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b5279690b643069d4dd544d2d32633" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 46049 } } peers { permanent_uuid: "c87accce8a18422aa2e7ac8e762869ba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36819 } } peers { permanent_uuid: "f2daa0b1b1004aeb9cc0d5d886d9512f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41469 } }
I20260926 15:00:14.229045 9646 raft_consensus.cc:374] T 1f377cd331c24e148fbcbc7085870572 P f2daa0b1b1004aeb9cc0d5d886d9512f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:14.229096 9646 raft_consensus.cc:729] T 1f377cd331c24e148fbcbc7085870572 P f2daa0b1b1004aeb9cc0d5d886d9512f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2daa0b1b1004aeb9cc0d5d886d9512f, State: Initialized, Role: FOLLOWER
I20260926 15:00:14.229245 9646 consensus_queue.cc:260] T 1f377cd331c24e148fbcbc7085870572 P f2daa0b1b1004aeb9cc0d5d886d9512f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b5279690b643069d4dd544d2d32633" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 46049 } } peers { permanent_uuid: "c87accce8a18422aa2e7ac8e762869ba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36819 } } peers { permanent_uuid: "f2daa0b1b1004aeb9cc0d5d886d9512f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41469 } }
I20260926 15:00:14.229573 9646 ts_tablet_manager.cc:1428] T 1f377cd331c24e148fbcbc7085870572 P f2daa0b1b1004aeb9cc0d5d886d9512f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:00:14.229717 9495 heartbeater.cc:503] Master 127.8.102.62:44219 was elected leader, sending a full tablet report...
I20260926 15:00:14.229811 9640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:14.213521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:36819 (local address 127.0.0.1:58824)
0926 15:00:14.214587 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:14.214598 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:14.214685 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:14.225037 (+ 10352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:14.225039 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:14.225044 (+ 5us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:14.225203 (+ 159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:14.225206 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.226887 (+ 1681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.226889 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:14.228296 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:14.228300 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.229529 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.229531 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:14.229599 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:14.229605 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 15:00:14.229664 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 15:00:14.229721 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1052,"thread_start_us":328,"threads_started":1}
I20260926 15:00:14.230083 9649 tablet_bootstrap.cc:492] T 1f377cd331c24e148fbcbc7085870572 P 74b5279690b643069d4dd544d2d32633: Bootstrap starting.
I20260926 15:00:14.230238 9645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:14.213730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58824 (local address 127.8.102.3:36819)
0926 15:00:14.219116 (+ 5386us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:14.219120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:14.219129 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:14.219168 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:14.219170 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:14.219183 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:14.219251 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:14.225308 (+ 6057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.226709 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.228435 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.229390 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.229507 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:14.229574 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:14.229708 (+ 134us) server_negotiation.cc:299] Negotiation successful
0926 15:00:14.229748 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5340,"thread_start_us":74,"threads_started":1}
I20260926 15:00:14.230901 9649 tablet_bootstrap.cc:654] T 1f377cd331c24e148fbcbc7085870572 P 74b5279690b643069d4dd544d2d32633: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:14.231280 9563 tablet_service.cc:1467] Processing CreateTablet for tablet 1f377cd331c24e148fbcbc7085870572 (DEFAULT_TABLE table=test-table [id=63bdfc04118340919d01095053dbbc9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:14.231499 9649 log.cc:824] T 1f377cd331c24e148fbcbc7085870572 P 74b5279690b643069d4dd544d2d32633: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:14.231669 9563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1f377cd331c24e148fbcbc7085870572. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:14.232343 9649 tablet_bootstrap.cc:492] T 1f377cd331c24e148fbcbc7085870572 P 74b5279690b643069d4dd544d2d32633: No bootstrap required, opened a new log
I20260926 15:00:14.232420 9649 ts_tablet_manager.cc:1397] T 1f377cd331c24e148fbcbc7085870572 P 74b5279690b643069d4dd544d2d32633: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 15:00:14.234529 9649 raft_consensus.cc:348] T 1f377cd331c24e148fbcbc7085870572 P 74b5279690b643069d4dd544d2d32633 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b5279690b643069d4dd544d2d32633" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 46049 } } peers { permanent_uuid: "c87accce8a18422aa2e7ac8e762869ba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36819 } } peers { permanent_uuid: "f2daa0b1b1004aeb9cc0d5d886d9512f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41469 } }
I20260926 15:00:14.234766 9649 raft_consensus.cc:374] T 1f377cd331c24e148fbcbc7085870572 P 74b5279690b643069d4dd544d2d32633 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:14.234807 9649 raft_consensus.cc:729] T 1f377cd331c24e148fbcbc7085870572 P 74b5279690b643069d4dd544d2d32633 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74b5279690b643069d4dd544d2d32633, State: Initialized, Role: FOLLOWER
I20260926 15:00:14.234931 9649 consensus_queue.cc:260] T 1f377cd331c24e148fbcbc7085870572 P 74b5279690b643069d4dd544d2d32633 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b5279690b643069d4dd544d2d32633" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 46049 } } peers { permanent_uuid: "c87accce8a18422aa2e7ac8e762869ba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36819 } } peers { permanent_uuid: "f2daa0b1b1004aeb9cc0d5d886d9512f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41469 } }
I20260926 15:00:14.235229 9649 ts_tablet_manager.cc:1428] T 1f377cd331c24e148fbcbc7085870572 P 74b5279690b643069d4dd544d2d32633: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 15:00:14.235224 9651 tablet_bootstrap.cc:492] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba: Bootstrap starting.
I20260926 15:00:14.235355 9361 heartbeater.cc:503] Master 127.8.102.62:44219 was elected leader, sending a full tablet report...
I20260926 15:00:14.235895 9651 tablet_bootstrap.cc:654] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:14.236505 9651 log.cc:824] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:14.237253 9651 tablet_bootstrap.cc:492] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba: No bootstrap required, opened a new log
I20260926 15:00:14.237339 9651 ts_tablet_manager.cc:1397] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:00:14.238988 9651 raft_consensus.cc:348] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b5279690b643069d4dd544d2d32633" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 46049 } } peers { permanent_uuid: "c87accce8a18422aa2e7ac8e762869ba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36819 } } peers { permanent_uuid: "f2daa0b1b1004aeb9cc0d5d886d9512f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41469 } }
I20260926 15:00:14.239131 9651 raft_consensus.cc:374] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:14.239153 9651 raft_consensus.cc:729] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c87accce8a18422aa2e7ac8e762869ba, State: Initialized, Role: FOLLOWER
I20260926 15:00:14.239236 9651 consensus_queue.cc:260] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b5279690b643069d4dd544d2d32633" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 46049 } } peers { permanent_uuid: "c87accce8a18422aa2e7ac8e762869ba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36819 } } peers { permanent_uuid: "f2daa0b1b1004aeb9cc0d5d886d9512f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41469 } }
I20260926 15:00:14.239494 9651 ts_tablet_manager.cc:1428] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:00:14.239491 9629 heartbeater.cc:503] Master 127.8.102.62:44219 was elected leader, sending a full tablet report...
W20260926 15:00:14.263265 9496 tablet.cc:2369] T 1f377cd331c24e148fbcbc7085870572 P f2daa0b1b1004aeb9cc0d5d886d9512f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:00:14.349941 9362 tablet.cc:2369] T 1f377cd331c24e148fbcbc7085870572 P 74b5279690b643069d4dd544d2d32633: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:00:14.435391 9630 tablet.cc:2369] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:00:14.518388 9654 raft_consensus.cc:482] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:14.518570 9654 raft_consensus.cc:504] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b5279690b643069d4dd544d2d32633" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 46049 } } peers { permanent_uuid: "c87accce8a18422aa2e7ac8e762869ba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36819 } } peers { permanent_uuid: "f2daa0b1b1004aeb9cc0d5d886d9512f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41469 } }
I20260926 15:00:14.518999 9654 leader_election.cc:288] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 74b5279690b643069d4dd544d2d32633 (127.8.102.1:46049), f2daa0b1b1004aeb9cc0d5d886d9512f (127.8.102.2:41469)
I20260926 15:00:14.523111 9644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:14.519277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:44443 (local address 127.8.102.2:41469)
0926 15:00:14.519340 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:14.519343 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:14.519354 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:00:14.519427 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:14.519428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:14.519449 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:14.519530 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:14.519694 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.520458 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.522064 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.522618 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.522727 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:14.522739 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:14.523005 (+ 266us) server_negotiation.cc:299] Negotiation successful
0926 15:00:14.523044 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 15:00:14.523111 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:14.519338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:46277 (local address 127.8.102.1:46049)
0926 15:00:14.519388 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:14.519390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:14.519516 (+ 126us) server_negotiation.cc:407] Connection header received
0926 15:00:14.519582 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:14.519583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:14.519600 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:14.519666 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:14.519792 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.520476 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.522063 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.522618 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.522726 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:14.522739 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:14.522992 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 15:00:14.523036 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 15:00:14.523177 9631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:14.519206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:41469 (local address 127.8.102.3:44443)
0926 15:00:14.519289 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:14.519301 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:14.519350 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:14.519537 (+ 187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:14.519538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:14.519540 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:14.519613 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:14.519615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.520587 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.520588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:14.521949 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:14.521950 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.522756 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.522758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:14.522920 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:14.522924 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:00:14.522986 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 15:00:14.523027 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":64,"mutex_wait_us":85}
I20260926 15:00:14.523187 9655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:14.519221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:46049 (local address 127.8.102.3:46277)
0926 15:00:14.519434 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:14.519445 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:14.519529 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:14.519663 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:14.519664 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:14.519666 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:14.519727 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:14.519728 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.520587 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.520588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:14.521944 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:14.521949 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.522756 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.522758 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:14.522875 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:14.522880 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:00:14.522964 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 15:00:14.523020 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":195,"mutex_wait_us":76,"thread_start_us":92,"threads_started":1}
I20260926 15:00:14.523567 9315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f377cd331c24e148fbcbc7085870572" candidate_uuid: "c87accce8a18422aa2e7ac8e762869ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74b5279690b643069d4dd544d2d32633" is_pre_election: true
I20260926 15:00:14.523567 9449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f377cd331c24e148fbcbc7085870572" candidate_uuid: "c87accce8a18422aa2e7ac8e762869ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2daa0b1b1004aeb9cc0d5d886d9512f" is_pre_election: true
I20260926 15:00:14.523770 9449 raft_consensus.cc:2454] T 1f377cd331c24e148fbcbc7085870572 P f2daa0b1b1004aeb9cc0d5d886d9512f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87accce8a18422aa2e7ac8e762869ba in term 0.
I20260926 15:00:14.523763 9315 raft_consensus.cc:2454] T 1f377cd331c24e148fbcbc7085870572 P 74b5279690b643069d4dd544d2d32633 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c87accce8a18422aa2e7ac8e762869ba in term 0.
I20260926 15:00:14.524042 9516 leader_election.cc:302] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c87accce8a18422aa2e7ac8e762869ba, f2daa0b1b1004aeb9cc0d5d886d9512f; no voters:
I20260926 15:00:14.524192 9654 raft_consensus.cc:2781] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:00:14.524250 9654 raft_consensus.cc:482] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:00:14.524281 9654 raft_consensus.cc:3037] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:14.524971 9654 raft_consensus.cc:504] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b5279690b643069d4dd544d2d32633" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 46049 } } peers { permanent_uuid: "c87accce8a18422aa2e7ac8e762869ba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36819 } } peers { permanent_uuid: "f2daa0b1b1004aeb9cc0d5d886d9512f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41469 } }
I20260926 15:00:14.525115 9654 leader_election.cc:288] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba [CANDIDATE]: Term 1 election: Requested vote from peers 74b5279690b643069d4dd544d2d32633 (127.8.102.1:46049), f2daa0b1b1004aeb9cc0d5d886d9512f (127.8.102.2:41469)
I20260926 15:00:14.525311 9315 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f377cd331c24e148fbcbc7085870572" candidate_uuid: "c87accce8a18422aa2e7ac8e762869ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74b5279690b643069d4dd544d2d32633"
I20260926 15:00:14.525306 9449 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1f377cd331c24e148fbcbc7085870572" candidate_uuid: "c87accce8a18422aa2e7ac8e762869ba" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f2daa0b1b1004aeb9cc0d5d886d9512f"
I20260926 15:00:14.525403 9449 raft_consensus.cc:3037] T 1f377cd331c24e148fbcbc7085870572 P f2daa0b1b1004aeb9cc0d5d886d9512f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:14.525403 9315 raft_consensus.cc:3037] T 1f377cd331c24e148fbcbc7085870572 P 74b5279690b643069d4dd544d2d32633 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:14.526357 9315 raft_consensus.cc:2454] T 1f377cd331c24e148fbcbc7085870572 P 74b5279690b643069d4dd544d2d32633 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87accce8a18422aa2e7ac8e762869ba in term 1.
I20260926 15:00:14.526357 9449 raft_consensus.cc:2454] T 1f377cd331c24e148fbcbc7085870572 P f2daa0b1b1004aeb9cc0d5d886d9512f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c87accce8a18422aa2e7ac8e762869ba in term 1.
I20260926 15:00:14.526547 9517 leader_election.cc:302] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 74b5279690b643069d4dd544d2d32633, c87accce8a18422aa2e7ac8e762869ba; no voters:
I20260926 15:00:14.526669 9654 raft_consensus.cc:2781] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:14.526855 9654 raft_consensus.cc:686] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba [term 1 LEADER]: Becoming Leader. State: Replica: c87accce8a18422aa2e7ac8e762869ba, State: Running, Role: LEADER
I20260926 15:00:14.526937 9654 consensus_queue.cc:237] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b5279690b643069d4dd544d2d32633" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 46049 } } peers { permanent_uuid: "c87accce8a18422aa2e7ac8e762869ba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36819 } } peers { permanent_uuid: "f2daa0b1b1004aeb9cc0d5d886d9512f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41469 } }
I20260926 15:00:14.527940 9169 catalog_manager.cc:5495] T 1f377cd331c24e148fbcbc7085870572 P c87accce8a18422aa2e7ac8e762869ba reported cstate change: term changed from 0 to 1, leader changed from <none> to c87accce8a18422aa2e7ac8e762869ba (127.8.102.3). New cstate: current_term: 1 leader_uuid: "c87accce8a18422aa2e7ac8e762869ba" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "74b5279690b643069d4dd544d2d32633" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 46049 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c87accce8a18422aa2e7ac8e762869ba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36819 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f2daa0b1b1004aeb9cc0d5d886d9512f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41469 } health_report { overall_health: UNKNOWN } } }
Sep 26 15:00:14 dist-test-slave-jnhs krb5kdc[9125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434813, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.1@KRBTEST.COM
I20260926 15:00:14.588325 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:14.580524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43790 (local address 127.8.102.1:46049)
0926 15:00:14.580596 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:14.580598 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:14.580752 (+ 154us) server_negotiation.cc:407] Connection header received
0926 15:00:14.580797 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:14.580797 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:14.580807 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:14.580867 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:14.581386 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.581884 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.582728 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.582907 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.582962 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:14.585389 (+ 2427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:14.585402 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:14.585402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:14.585410 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:14.587185 (+ 1775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:14.587565 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:14.587568 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:14.587569 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:14.587630 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:14.587849 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:14.587849 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:14.587850 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:14.587988 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:14.588089 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:14.588167 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 15:00:14.588227 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 15:00:14 dist-test-slave-jnhs krb5kdc[9125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434814, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 15:00:14 dist-test-slave-jnhs krb5kdc[9125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434814, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.102.1@KRBTEST.COM
I20260926 15:00:14.612318 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:14.604809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.8.102.1:46049)
0926 15:00:14.604874 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:14.604876 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:14.604958 (+ 82us) server_negotiation.cc:407] Connection header received
0926 15:00:14.605012 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:14.605013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:14.605023 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:14.605073 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:14.605622 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.606109 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.606902 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.607079 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.607133 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:14.609581 (+ 2448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:14.609583 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:14.609583 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:14.609590 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:14.611061 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:14.611504 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:14.611505 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:14.611506 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:14.611567 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:14.611767 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:14.611768 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:14.611768 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:14.611878 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:14.611964 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:14.612174 (+ 210us) server_negotiation.cc:299] Negotiation successful
0926 15:00:14.612244 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 15:00:14 dist-test-slave-jnhs krb5kdc[9125](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 15:00:14 dist-test-slave-jnhs krb5kdc[9125](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434814, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.1@KRBTEST.COM
I20260926 15:00:14.635469 9642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:14.627843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43820 (local address 127.8.102.1:46049)
0926 15:00:14.627971 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:14.627974 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:14.628084 (+ 110us) server_negotiation.cc:407] Connection header received
0926 15:00:14.628192 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:14.628193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:14.628203 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:14.628277 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:14.628848 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.629378 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.630181 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:14.630363 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:14.630426 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:14.632976 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:14.632977 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:14.632978 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:14.632984 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:14.634437 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:14.634837 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:14.634838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:14.634838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:14.634891 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:14.635104 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:14.635104 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:14.635104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:14.635188 (+ 84us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:14.635254 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:14.635349 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 15:00:14.635404 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260926 15:00:14.636008 9275 tablet_service.cc:2908] Rejecting scan request for tablet 1f377cd331c24e148fbcbc7085870572: Uninitialized: clean time has not yet been initialized
I20260926 15:00:14.636930 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 9232
I20260926 15:00:14.642004 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 9366
I20260926 15:00:14.646901 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 9500
I20260926 15:00:14.651433 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 9138
2026-09-26T15:00:14Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (2294 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:00:14 dist-test-slave-jnhs krb5kdc[9670](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:00:14 dist-test-slave-jnhs krb5kdc[9670](info): set up 2 sockets
Sep 26 15:00:14 dist-test-slave-jnhs krb5kdc[9670](info): commencing operation
krb5kdc: starting...
W20260926 15:00:15.702345 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.030s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:00:15 dist-test-slave-jnhs krb5kdc[9670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:00:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:00:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:00:15.825467 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:36745
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:36745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:15.907881 9683 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:15.908159 9683 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:15.908200 9683 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:15.909463 9683 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:15.909526 9683 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 15:00:15.909547 9683 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:15.909570 9683 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:15.909603 9683 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:15.909626 9683 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:00:15.911170 9683 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:36745
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:36745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9683
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:15.911480 9683 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:15.911726 9683 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:15.916980 9691 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:15.917057 9683 server_base.cc:1034] running on GCE node
W20260926 15:00:15.917016 9689 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:15.917011 9688 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:15.917523 9683 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:15.917845 9683 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:15.918987 9683 hybrid_clock.cc:648] HybridClock initialized: now 1790434815918966 us; error 39 us; skew 500 ppm
Sep 26 15:00:15 dist-test-slave-jnhs krb5kdc[9670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434815, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:15.927850 9683 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:00:15.928385 9683 webserver.cc:460] Webserver started at http://127.8.102.62:36899/ using document root <none> and password file <none>
I20260926 15:00:15.928639 9683 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:15.928711 9683 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:15.928874 9683 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:15.930090 9683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "1ff0f95a826245dcae03caf1969e076f"
format_stamp: "Formatted at 2026-09-26 15:00:15 on dist-test-slave-jnhs"
server_key: "874583685677e419c19f63db49ac4d23"
server_key_iv: "51d1edc6a7a857d63ce66bcb95ef3da7"
server_key_version: "encryptionkey@0"
I20260926 15:00:15.930496 9683 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "1ff0f95a826245dcae03caf1969e076f"
format_stamp: "Formatted at 2026-09-26 15:00:15 on dist-test-slave-jnhs"
server_key: "874583685677e419c19f63db49ac4d23"
server_key_iv: "51d1edc6a7a857d63ce66bcb95ef3da7"
server_key_version: "encryptionkey@0"
I20260926 15:00:15.932018 9683 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:15.932947 9698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:15.933153 9683 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:15.933264 9683 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "1ff0f95a826245dcae03caf1969e076f"
format_stamp: "Formatted at 2026-09-26 15:00:15 on dist-test-slave-jnhs"
server_key: "874583685677e419c19f63db49ac4d23"
server_key_iv: "51d1edc6a7a857d63ce66bcb95ef3da7"
server_key_version: "encryptionkey@0"
I20260926 15:00:15.933352 9683 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:15.942911 9683 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:15.944947 9683 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:15.945113 9683 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:15.949908 9683 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:36745
I20260926 15:00:15.949961 9760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:36745 every 8 connection(s)
I20260926 15:00:15.950297 9683 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:00:15.950487 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 9683
I20260926 15:00:15.950628 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:00:15.950807 8600 external_mini_cluster.cc:1428] Setting key ad6fa9427c5dce33ebb549f163866709
I20260926 15:00:15.951090 9761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:15.954484 9761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f: Bootstrap starting.
I20260926 15:00:15.955171 9761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:15.955498 9761 log.cc:824] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:15.956223 9761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f: No bootstrap required, opened a new log
Sep 26 15:00:15 dist-test-slave-jnhs krb5kdc[9670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:15.958187 9761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff0f95a826245dcae03caf1969e076f" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36745 } }
I20260926 15:00:15.958354 9761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:15.958401 9761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ff0f95a826245dcae03caf1969e076f, State: Initialized, Role: FOLLOWER
I20260926 15:00:15.958536 9761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff0f95a826245dcae03caf1969e076f" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36745 } }
I20260926 15:00:15.958675 9761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:00:15.958726 9761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:00:15.958781 9761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:15.959445 9761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff0f95a826245dcae03caf1969e076f" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36745 } }
I20260926 15:00:15.959568 9761 leader_election.cc:302] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1ff0f95a826245dcae03caf1969e076f; no voters:
I20260926 15:00:15.959769 9761 leader_election.cc:288] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:00:15.959828 9766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:15.960026 9766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f [term 1 LEADER]: Becoming Leader. State: Replica: 1ff0f95a826245dcae03caf1969e076f, State: Running, Role: LEADER
I20260926 15:00:15.960045 9761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:15.960181 9766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff0f95a826245dcae03caf1969e076f" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36745 } }
I20260926 15:00:15.960596 9766 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1ff0f95a826245dcae03caf1969e076f. Latest consensus state: current_term: 1 leader_uuid: "1ff0f95a826245dcae03caf1969e076f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff0f95a826245dcae03caf1969e076f" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36745 } } }
I20260926 15:00:15.960606 9767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1ff0f95a826245dcae03caf1969e076f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1ff0f95a826245dcae03caf1969e076f" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36745 } } }
I20260926 15:00:15.960774 9766 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:15.960788 9767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:15.961025 9775 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:00:15.961612 9775 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:00:15.962674 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:15.951370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51612 (local address 127.8.102.62:36745)
0926 15:00:15.951565 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:15.951575 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:15.951610 (+ 35us) server_negotiation.cc:407] Connection header received
0926 15:00:15.952245 (+ 635us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:15.952254 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:15.952324 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:15.952483 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:15.953090 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:15.954502 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:15.955366 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:15.955868 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:15.955991 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:15.958314 (+ 2323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:15.958318 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:15.958319 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:15.958338 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:15.960515 (+ 2177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:15.960976 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:15.960979 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:15.960979 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:15.961100 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:15.961326 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:15.961326 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:15.961327 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:15.961632 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:15.962244 (+ 612us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:15.962317 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 15:00:15.962457 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":90,"threads_started":1}
I20260926 15:00:15.963487 9775 catalog_manager.cc:1347] Generated new cluster ID: 31b724623c6b4821ab216ebf8e65ae59
I20260926 15:00:15.963543 9775 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:00:15.968492 9775 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:00:15.969046 9775 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:00:15.974207 9775 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f: Generated new TSK 0
I20260926 15:00:15.974402 9775 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
I20260926 15:00:16.006309 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36745
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:16.111061 9787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:16.111269 9787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:16.111296 9787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:16.112586 9787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:16.112636 9787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:16.112658 9787 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:00:16.112675 9787 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:00:16.114336 9787 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.102.62:36745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9787
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:16.114625 9787 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:16.114929 9787 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:16.121248 9793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:16.121285 9795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:16.121343 9787 server_base.cc:1034] running on GCE node
W20260926 15:00:16.121285 9792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:16.121610 9787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:16.121899 9787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:16.123075 9787 hybrid_clock.cc:648] HybridClock initialized: now 1790434816123061 us; error 47 us; skew 500 ppm
Sep 26 15:00:16 dist-test-slave-jnhs krb5kdc[9670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434816, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:16.125711 9787 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:00:16.126175 9787 webserver.cc:460] Webserver started at http://127.8.102.1:43603/ using document root <none> and password file <none>
I20260926 15:00:16.126435 9787 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:16.126483 9787 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:16.126614 9787 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:16.127763 9787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "b3a929efe2d84366ac279ae7d3056efa"
format_stamp: "Formatted at 2026-09-26 15:00:16 on dist-test-slave-jnhs"
server_key: "aef948cb5d822914610dc0f67c567129"
server_key_iv: "7a2e1d147940263074dc172c331287cf"
server_key_version: "encryptionkey@0"
I20260926 15:00:16.128120 9787 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "b3a929efe2d84366ac279ae7d3056efa"
format_stamp: "Formatted at 2026-09-26 15:00:16 on dist-test-slave-jnhs"
server_key: "aef948cb5d822914610dc0f67c567129"
server_key_iv: "7a2e1d147940263074dc172c331287cf"
server_key_version: "encryptionkey@0"
I20260926 15:00:16.129515 9787 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 15:00:16.130388 9802 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:16.130584 9787 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:16.130724 9787 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "b3a929efe2d84366ac279ae7d3056efa"
format_stamp: "Formatted at 2026-09-26 15:00:16 on dist-test-slave-jnhs"
server_key: "aef948cb5d822914610dc0f67c567129"
server_key_iv: "7a2e1d147940263074dc172c331287cf"
server_key_version: "encryptionkey@0"
I20260926 15:00:16.130807 9787 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:16.149371 9787 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:16.151906 9787 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:16.152086 9787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:16.152647 9787 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:16.152715 9787 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:16.152761 9787 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:16.152804 9787 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:16.159899 9787 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:39659
I20260926 15:00:16.160080 9917 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:39659 every 8 connection(s)
I20260926 15:00:16.160450 9787 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
Sep 26 15:00:16 dist-test-slave-jnhs krb5kdc[9670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434816, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:16.162439 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 9787
I20260926 15:00:16.162549 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:00:16.162760 8600 external_mini_cluster.cc:1428] Setting key 84d362e177a8033e4b27eadc567c5b03
I20260926 15:00:16.164712 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.153278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:38863 (local address 127.8.102.62:36745)
0926 15:00:16.153332 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:16.153334 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:16.153601 (+ 267us) server_negotiation.cc:407] Connection header received
0926 15:00:16.154203 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:16.154204 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:16.154219 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:16.154296 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:16.155363 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.156234 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.158007 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.158364 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.158469 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.161487 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:16.161489 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:16.161489 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:16.161496 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:16.163076 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:16.163638 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:16.163639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:16.163639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:16.163711 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:16.163999 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:16.164000 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:16.164000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:16.164129 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:16.164201 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:16.164517 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 15:00:16.164615 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 15:00:16.164907 9811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.153244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36745 (local address 127.8.102.1:38863)
0926 15:00:16.153413 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:16.153489 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:16.154041 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:16.154392 (+ 351us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:16.154398 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:16.154911 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:16.155237 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.155243 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.156368 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.156369 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.157874 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.157878 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.158474 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.158475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.158638 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.159344 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:16.159355 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:16.161396 (+ 2041us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:16.163182 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:16.163188 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:16.163190 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:16.163547 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:16.163778 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:16.163779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:16.163779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:16.163913 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:16.164205 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:16.164212 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:16.164362 (+ 150us) client_negotiation.cc:769] Sending connection context
0926 15:00:16.164492 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 15:00:16.164665 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":98,"threads_started":1}
I20260926 15:00:16.165499 9918 heartbeater.cc:347] Connected to a master server at 127.8.102.62:36745
I20260926 15:00:16.165611 9918 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:16.165803 9918 heartbeater.cc:511] Master 127.8.102.62:36745 requested a full tablet report, sending...
I20260926 15:00:16.166383 9715 ts_manager.cc:194] Registered new tserver with Master: b3a929efe2d84366ac279ae7d3056efa (127.8.102.1:39659)
I20260926 15:00:16.167241 9715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:38863
WARNING: no policy specified for kudu/127.8.102.2@KRBTEST.COM; defaulting to no policy
I20260926 15:00:16.174582 9925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.166837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36745 (local address 127.8.102.1:41593)
0926 15:00:16.167511 (+ 674us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:16.167522 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:16.167605 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:16.167774 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:16.167775 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:16.167976 (+ 201us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:16.168044 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.168046 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.168879 (+ 833us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.168880 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.169837 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.169840 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.170245 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.170246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.170331 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.171135 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:16.171141 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:16.171578 (+ 437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:16.173410 (+ 1832us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:16.173411 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:16.173412 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:16.173656 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:16.173882 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:16.173883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:16.173883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:16.173950 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:16.174245 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:16.174245 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:16.174291 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 15:00:16.174348 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 15:00:16.174470 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":70,"threads_started":1}
I20260926 15:00:16.174635 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.166896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:41593 (local address 127.8.102.62:36745)
0926 15:00:16.166935 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:16.166937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:16.167592 (+ 655us) server_negotiation.cc:407] Connection header received
0926 15:00:16.167659 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:16.167660 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:16.167670 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:16.167717 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:16.168139 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.168779 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.169948 (+ 1169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.170155 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.170223 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.171696 (+ 1473us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:16.171698 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:16.171699 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:16.171708 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:16.173306 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:16.173732 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:16.173733 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:16.173734 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:16.173792 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:16.174047 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:16.174048 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:16.174048 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:16.174174 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:16.174237 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:16.174363 (+ 126us) server_negotiation.cc:299] Negotiation successful
0926 15:00:16.174551 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
I20260926 15:00:16.193190 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36745
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:16.278277 9927 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:16.278522 9927 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:16.278563 9927 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:16.279805 9927 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:16.279884 9927 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:16.279917 9927 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:00:16.279956 9927 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:00:16.281478 9927 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.102.62:36745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.9927
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:16.281796 9927 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:16.282043 9927 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:16.287322 9933 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:16.287343 9935 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:16.287454 9927 server_base.cc:1034] running on GCE node
W20260926 15:00:16.287536 9932 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:16.287976 9927 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:16.288264 9927 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:16.289417 9927 hybrid_clock.cc:648] HybridClock initialized: now 1790434816289401 us; error 31 us; skew 500 ppm
Sep 26 15:00:16 dist-test-slave-jnhs krb5kdc[9670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434816, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:16.292021 9927 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:00:16.292520 9927 webserver.cc:460] Webserver started at http://127.8.102.2:38283/ using document root <none> and password file <none>
I20260926 15:00:16.292770 9927 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:16.292840 9927 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:16.292999 9927 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:16.294070 9927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "09fbced4b74c49778510ddf516a46773"
format_stamp: "Formatted at 2026-09-26 15:00:16 on dist-test-slave-jnhs"
server_key: "1f389e535a35a54a3a2d448333b7a987"
server_key_iv: "fa98d1575484ddcc6fbe1458b6688f12"
server_key_version: "encryptionkey@0"
I20260926 15:00:16.294463 9927 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "09fbced4b74c49778510ddf516a46773"
format_stamp: "Formatted at 2026-09-26 15:00:16 on dist-test-slave-jnhs"
server_key: "1f389e535a35a54a3a2d448333b7a987"
server_key_iv: "fa98d1575484ddcc6fbe1458b6688f12"
server_key_version: "encryptionkey@0"
I20260926 15:00:16.295935 9927 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:16.296819 9942 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:16.297025 9927 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:16.297127 9927 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "09fbced4b74c49778510ddf516a46773"
format_stamp: "Formatted at 2026-09-26 15:00:16 on dist-test-slave-jnhs"
server_key: "1f389e535a35a54a3a2d448333b7a987"
server_key_iv: "fa98d1575484ddcc6fbe1458b6688f12"
server_key_version: "encryptionkey@0"
I20260926 15:00:16.297224 9927 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:16.309660 9927 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:16.311933 9927 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:16.312211 9927 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:16.312858 9927 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:16.312915 9927 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:16.312954 9927 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:16.312968 9927 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:00:16 dist-test-slave-jnhs krb5kdc[9670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434816, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:16.320547 9927 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:39751
I20260926 15:00:16.320919 9927 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:00:16.320926 10057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:39751 every 8 connection(s)
I20260926 15:00:16.323654 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.313328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:39353 (local address 127.8.102.62:36745)
0926 15:00:16.313386 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:16.313389 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:16.313726 (+ 337us) server_negotiation.cc:407] Connection header received
0926 15:00:16.314261 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:16.314263 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:16.314277 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:16.314343 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:16.315136 (+ 793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.315713 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.317034 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.317332 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.317425 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.320236 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:16.320238 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:16.320238 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:16.320246 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:16.321933 (+ 1687us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:16.322569 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:16.322571 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:16.322571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:16.322676 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:16.322928 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:16.322929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:16.322930 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:16.323058 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:16.323138 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:16.323416 (+ 278us) server_negotiation.cc:299] Negotiation successful
0926 15:00:16.323556 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 15:00:16.323799 9951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.313329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36745 (local address 127.8.102.2:39353)
0926 15:00:16.313565 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:16.313619 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:16.314138 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:16.314427 (+ 289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:16.314431 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:16.314818 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:16.315028 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.315034 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.315823 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.315824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.316925 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.316927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.317447 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.317448 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.317581 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.318197 (+ 616us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:16.318208 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:16.320081 (+ 1873us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:16.322023 (+ 1942us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:16.322080 (+ 57us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:16.322082 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:16.322480 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:16.322732 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:16.322732 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:16.322733 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:16.322848 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:16.323129 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:16.323135 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:16.323282 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 15:00:16.323396 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 15:00:16.323567 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":119,"threads_started":1}
I20260926 15:00:16.324390 10058 heartbeater.cc:347] Connected to a master server at 127.8.102.62:36745
I20260926 15:00:16.324509 10058 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:16.324694 10058 heartbeater.cc:511] Master 127.8.102.62:36745 requested a full tablet report, sending...
I20260926 15:00:16.325145 9715 ts_manager.cc:194] Registered new tserver with Master: 09fbced4b74c49778510ddf516a46773 (127.8.102.2:39751)
I20260926 15:00:16.325640 9715 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:39353
I20260926 15:00:16.328307 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 9927
I20260926 15:00:16.328433 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:00:16.328622 8600 external_mini_cluster.cc:1428] Setting key 3512b479701f8f6010076ea9199d83ad
I20260926 15:00:16.331828 10064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.325407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36745 (local address 127.8.102.2:33791)
0926 15:00:16.325554 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:16.325566 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:16.325629 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:16.325745 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:16.325746 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:16.325951 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:16.326025 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.326027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.326932 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.326933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.327498 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.327500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.327905 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.327906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.327971 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.328517 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:16.328522 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:16.328833 (+ 311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:16.330597 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:16.330598 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:16.330598 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:16.330879 (+ 281us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:16.331139 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:16.331140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:16.331140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:16.331205 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:16.331560 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:16.331561 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:16.331604 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 15:00:16.331661 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 15:00:16.331751 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
I20260926 15:00:16.331969 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.325492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:33791 (local address 127.8.102.62:36745)
0926 15:00:16.325547 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:16.325548 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:16.325619 (+ 71us) server_negotiation.cc:407] Connection header received
0926 15:00:16.325685 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:16.325686 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:16.325694 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:16.325738 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:16.326099 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.326829 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.327606 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.327810 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.327879 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.328946 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:16.328948 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:16.328948 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:16.328956 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:16.330479 (+ 1523us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:16.330966 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:16.330967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:16.330968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:16.331047 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:16.331276 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:16.331277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:16.331277 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:16.331490 (+ 213us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:16.331562 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:16.331708 (+ 146us) server_negotiation.cc:299] Negotiation successful
0926 15:00:16.331821 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
WARNING: no policy specified for kudu/127.8.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
I20260926 15:00:16.358079 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36745
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:16.439844 10067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:16.440057 10067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:16.440081 10067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:16.441268 10067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:16.441314 10067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:16.441335 10067 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:00:16.441396 10067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:00:16.442884 10067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.102.62:36745
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10067
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:16.443141 10067 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:16.443380 10067 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:16.448432 10072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:16.448421 10075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:16.448424 10073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:16.448693 10067 server_base.cc:1034] running on GCE node
I20260926 15:00:16.448920 10067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:16.449225 10067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:16.450435 10067 hybrid_clock.cc:648] HybridClock initialized: now 1790434816450419 us; error 26 us; skew 500 ppm
Sep 26 15:00:16 dist-test-slave-jnhs krb5kdc[9670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434816, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:16.453200 10067 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:00:16.453600 10067 webserver.cc:460] Webserver started at http://127.8.102.3:32919/ using document root <none> and password file <none>
I20260926 15:00:16.453850 10067 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:16.453897 10067 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:16.454036 10067 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:16.455240 10067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "bfe8f417978b47e69d676c497d582a44"
format_stamp: "Formatted at 2026-09-26 15:00:16 on dist-test-slave-jnhs"
server_key: "fd8fba9f54c21530d6f10ff5f24ab51c"
server_key_iv: "b420c3652be511a6d4e6b1e535ddde11"
server_key_version: "encryptionkey@0"
I20260926 15:00:16.455614 10067 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "bfe8f417978b47e69d676c497d582a44"
format_stamp: "Formatted at 2026-09-26 15:00:16 on dist-test-slave-jnhs"
server_key: "fd8fba9f54c21530d6f10ff5f24ab51c"
server_key_iv: "b420c3652be511a6d4e6b1e535ddde11"
server_key_version: "encryptionkey@0"
I20260926 15:00:16.456996 10067 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:00:16.457813 10082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:16.458015 10067 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:16.458101 10067 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "bfe8f417978b47e69d676c497d582a44"
format_stamp: "Formatted at 2026-09-26 15:00:16 on dist-test-slave-jnhs"
server_key: "fd8fba9f54c21530d6f10ff5f24ab51c"
server_key_iv: "b420c3652be511a6d4e6b1e535ddde11"
server_key_version: "encryptionkey@0"
I20260926 15:00:16.458163 10067 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:16.464957 10067 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:16.467336 10067 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:16.467454 10067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:16.468041 10067 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:16.468082 10067 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:16.468106 10067 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:16.468123 10067 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:16.475227 10067 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:34639
I20260926 15:00:16.475303 10197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:34639 every 8 connection(s)
Sep 26 15:00:16 dist-test-slave-jnhs krb5kdc[9670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434816, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:16.475673 10067 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:00:16.479020 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.468641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:39999 (local address 127.8.102.62:36745)
0926 15:00:16.468698 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:16.468700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:16.468983 (+ 283us) server_negotiation.cc:407] Connection header received
0926 15:00:16.469575 (+ 592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:16.469576 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:16.469591 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:16.469677 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:16.470571 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.471462 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.472827 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.473065 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.473224 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.476029 (+ 2805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:16.476032 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:16.476032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:16.476042 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:16.477498 (+ 1456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:16.477937 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:16.477938 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:16.477939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:16.478003 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:16.478270 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:16.478271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:16.478271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:16.478398 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:16.478480 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:16.478842 (+ 362us) server_negotiation.cc:299] Negotiation successful
0926 15:00:16.478925 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 15:00:16.479230 10091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.468643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36745 (local address 127.8.102.3:39999)
0926 15:00:16.468843 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:16.468879 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:16.469418 (+ 539us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:16.469766 (+ 348us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:16.469772 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:16.470159 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:16.470456 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.470463 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.471591 (+ 1128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.471592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.472642 (+ 1050us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.472645 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.473239 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.473240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.473389 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.473984 (+ 595us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:16.473995 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:16.475916 (+ 1921us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:16.477610 (+ 1694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:16.477616 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:16.477618 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:16.477851 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:16.478073 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:16.478073 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:16.478074 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:16.478196 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:16.478470 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:16.478476 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:16.478625 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 15:00:16.478818 (+ 193us) client_negotiation.cc:240] Negotiation successful
0926 15:00:16.478985 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":111,"threads_started":1}
I20260926 15:00:16.479786 10198 heartbeater.cc:347] Connected to a master server at 127.8.102.62:36745
I20260926 15:00:16.479869 10198 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:16.480084 10198 heartbeater.cc:511] Master 127.8.102.62:36745 requested a full tablet report, sending...
I20260926 15:00:16.480532 9714 ts_manager.cc:194] Registered new tserver with Master: bfe8f417978b47e69d676c497d582a44 (127.8.102.3:34639)
I20260926 15:00:16.480945 9714 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:39999
I20260926 15:00:16.483039 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 10067
I20260926 15:00:16.483124 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:00:16.483341 8600 external_mini_cluster.cc:1428] Setting key d7a590b57ee83f1afcdb25dfd8609f36
I20260926 15:00:16.484256 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:00:16.486820 10204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.480876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36745 (local address 127.8.102.3:36261)
0926 15:00:16.480996 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:16.481008 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:16.481085 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:16.481230 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:16.481231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:16.481406 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:16.481466 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.481467 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.482115 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.482116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.482629 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.482676 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.483047 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.483048 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.483126 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.483687 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:16.483693 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:16.484070 (+ 377us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:16.485728 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:16.485729 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:16.485730 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:16.485955 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:16.486179 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:16.486180 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:16.486180 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:16.486233 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:16.486501 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:16.486502 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:16.486541 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 15:00:16.486590 (+ 49us) client_negotiation.cc:240] Negotiation successful
0926 15:00:16.486743 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":67,"threads_started":1}
I20260926 15:00:16.486868 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.480950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:36261 (local address 127.8.102.62:36745)
0926 15:00:16.480999 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:16.481001 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:16.481061 (+ 60us) server_negotiation.cc:407] Connection header received
0926 15:00:16.481161 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:16.481162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:16.481173 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:16.481217 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:16.481544 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.482019 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.482753 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.482960 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.483046 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.484174 (+ 1128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:16.484175 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:16.484176 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:16.484183 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:16.485609 (+ 1426us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:16.486027 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:16.486028 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:16.486028 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:16.486102 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:16.486292 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:16.486292 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:16.486293 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:16.486439 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:16.486509 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:16.486618 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 15:00:16.486742 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
Sep 26 15:00:16 dist-test-slave-jnhs krb5kdc[9670](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 15:00:16 dist-test-slave-jnhs krb5kdc[9670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:16.507234 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.500173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51624 (local address 127.8.102.62:36745)
0926 15:00:16.500247 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:16.500248 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:16.500259 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:00:16.500301 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:16.500301 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:16.500311 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:16.500354 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:16.501013 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.501504 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.502264 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.502442 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.502497 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.504737 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:16.504739 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:16.504740 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:16.504746 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:16.506052 (+ 1306us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:16.506399 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:16.506400 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:16.506401 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:16.506451 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:16.506625 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:16.506625 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:16.506625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:16.506799 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:16.506885 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:16.507072 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 15:00:16.507133 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:00:16.509016 9714 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:00:16.510615 9714 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:00:16.517395 10219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.513242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:34639 (local address 127.0.0.1:40238)
0926 15:00:16.513603 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:16.513625 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:16.513711 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:16.513990 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:16.513993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:16.513995 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:16.514084 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.514087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.515069 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.515071 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.516078 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.516080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.517067 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.517068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.517158 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.517172 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 15:00:16.517248 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 15:00:16.517304 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":347,"mutex_wait_us":4,"thread_start_us":61,"threads_started":1}
I20260926 15:00:16.517421 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.513333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40238 (local address 127.8.102.3:34639)
0926 15:00:16.513560 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:16.513565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:16.513685 (+ 120us) server_negotiation.cc:407] Connection header received
0926 15:00:16.513816 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:16.513819 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:16.513835 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:16.513890 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:16.514166 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.514985 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.516182 (+ 1197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.516971 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.517049 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.517106 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:16.517271 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 15:00:16.517307 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":101,"threads_started":1}
I20260926 15:00:16.517632 10216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.513449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:39751 (local address 127.0.0.1:50584)
0926 15:00:16.513668 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:16.513680 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:16.513738 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:16.514078 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:16.514079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:16.514081 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:16.514161 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.514163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.515092 (+ 929us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.515093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.516310 (+ 1217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.516313 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.517224 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.517225 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.517281 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.517285 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:00:16.517330 (+ 45us) client_negotiation.cc:240] Negotiation successful
0926 15:00:16.517363 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":25,"thread_start_us":68,"threads_started":1}
I20260926 15:00:16.517961 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.513655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50584 (local address 127.8.102.2:39751)
0926 15:00:16.513859 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:16.513863 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:16.513872 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:16.513913 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:16.513915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:16.513931 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:16.514009 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:16.514238 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.514992 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.516413 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.517064 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.517134 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.517190 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:16.517433 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 15:00:16.517880 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":78,"threads_started":1}
I20260926 15:00:16.518146 10132 tablet_service.cc:1467] Processing CreateTablet for tablet 9bdcec4a97d14b4fbc6bfad9ef14d43f (DEFAULT_TABLE table=test-table [id=6e0fbad878ca46c3963ba070f0fdcbb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:16.518513 10215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.513241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:39659 (local address 127.0.0.1:56986)
0926 15:00:16.513579 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:16.513625 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:16.513703 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:16.513990 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:16.513993 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:16.513997 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:16.514084 (+ 87us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.514087 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.516295 (+ 2208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.516296 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.517365 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.517368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.518270 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.518271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.518338 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.518342 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:00:16.518403 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 15:00:16.518440 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":13,"spinlock_wait_cycles":768,"thread_start_us":97,"threads_started":1}
I20260926 15:00:16.518539 10132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bdcec4a97d14b4fbc6bfad9ef14d43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:16.518560 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.513421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56986 (local address 127.8.102.1:39659)
0926 15:00:16.513560 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:16.513565 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:16.513795 (+ 230us) server_negotiation.cc:407] Connection header received
0926 15:00:16.513838 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:16.513840 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:16.513857 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:16.513924 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:16.515098 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.516213 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.517467 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.518194 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.518271 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.518329 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:16.518426 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 15:00:16.518468 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":75,"threads_started":1}
I20260926 15:00:16.518585 9992 tablet_service.cc:1467] Processing CreateTablet for tablet 9bdcec4a97d14b4fbc6bfad9ef14d43f (DEFAULT_TABLE table=test-table [id=6e0fbad878ca46c3963ba070f0fdcbb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:16.518976 9992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bdcec4a97d14b4fbc6bfad9ef14d43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:16.519182 9852 tablet_service.cc:1467] Processing CreateTablet for tablet 9bdcec4a97d14b4fbc6bfad9ef14d43f (DEFAULT_TABLE table=test-table [id=6e0fbad878ca46c3963ba070f0fdcbb6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:16.519479 9852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9bdcec4a97d14b4fbc6bfad9ef14d43f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:16.521076 10221 tablet_bootstrap.cc:492] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44: Bootstrap starting.
I20260926 15:00:16.522106 10221 tablet_bootstrap.cc:654] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:16.522434 10221 log.cc:824] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:16.522722 10222 tablet_bootstrap.cc:492] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773: Bootstrap starting.
I20260926 15:00:16.522981 10224 tablet_bootstrap.cc:492] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa: Bootstrap starting.
I20260926 15:00:16.523280 10221 tablet_bootstrap.cc:492] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44: No bootstrap required, opened a new log
I20260926 15:00:16.523360 10221 ts_tablet_manager.cc:1397] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:00:16.523430 10222 tablet_bootstrap.cc:654] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:16.523641 10224 tablet_bootstrap.cc:654] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:16.523765 10222 log.cc:824] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:16.523874 10224 log.cc:824] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:16.524612 10222 tablet_bootstrap.cc:492] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773: No bootstrap required, opened a new log
I20260926 15:00:16.524662 10222 ts_tablet_manager.cc:1397] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:00:16.524793 10224 tablet_bootstrap.cc:492] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa: No bootstrap required, opened a new log
I20260926 15:00:16.524855 10224 ts_tablet_manager.cc:1397] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:00:16.525605 10221 raft_consensus.cc:348] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } }
I20260926 15:00:16.525774 10221 raft_consensus.cc:374] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:16.525807 10221 raft_consensus.cc:729] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe8f417978b47e69d676c497d582a44, State: Initialized, Role: FOLLOWER
I20260926 15:00:16.525929 10221 consensus_queue.cc:260] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } }
I20260926 15:00:16.526214 10221 ts_tablet_manager.cc:1428] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:00:16.526247 10198 heartbeater.cc:503] Master 127.8.102.62:36745 was elected leader, sending a full tablet report...
I20260926 15:00:16.526211 10222 raft_consensus.cc:348] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } }
I20260926 15:00:16.526364 10222 raft_consensus.cc:374] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:16.526404 10222 raft_consensus.cc:729] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09fbced4b74c49778510ddf516a46773, State: Initialized, Role: FOLLOWER
I20260926 15:00:16.526504 10222 consensus_queue.cc:260] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } }
I20260926 15:00:16.526818 10222 ts_tablet_manager.cc:1428] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:00:16.526881 10058 heartbeater.cc:503] Master 127.8.102.62:36745 was elected leader, sending a full tablet report...
I20260926 15:00:16.526921 10224 raft_consensus.cc:348] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } }
I20260926 15:00:16.527140 10224 raft_consensus.cc:374] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:16.527237 10224 raft_consensus.cc:729] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3a929efe2d84366ac279ae7d3056efa, State: Initialized, Role: FOLLOWER
I20260926 15:00:16.527393 10224 consensus_queue.cc:260] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } }
I20260926 15:00:16.527714 10224 ts_tablet_manager.cc:1428] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:00:16.527860 9918 heartbeater.cc:503] Master 127.8.102.62:36745 was elected leader, sending a full tablet report...
W20260926 15:00:16.571786 10059 tablet.cc:2369] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:00:16.661329 9919 tablet.cc:2369] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:00:16.726362 10199 tablet.cc:2369] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:00:16.843166 10227 raft_consensus.cc:482] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:16.843312 10227 raft_consensus.cc:504] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } }
I20260926 15:00:16.843619 10227 leader_election.cc:288] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3a929efe2d84366ac279ae7d3056efa (127.8.102.1:39659), 09fbced4b74c49778510ddf516a46773 (127.8.102.2:39751)
I20260926 15:00:16.845953 10228 raft_consensus.cc:482] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:16.846076 10228 raft_consensus.cc:504] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } }
I20260926 15:00:16.846843 10228 leader_election.cc:288] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers bfe8f417978b47e69d676c497d582a44 (127.8.102.3:34639), b3a929efe2d84366ac279ae7d3056efa (127.8.102.1:39659)
I20260926 15:00:16.847131 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.844112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:46677 (local address 127.8.102.2:39751)
0926 15:00:16.844167 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:16.844169 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:16.844272 (+ 103us) server_negotiation.cc:407] Connection header received
0926 15:00:16.844302 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:16.844303 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:16.844316 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:16.844391 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:16.844554 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.845107 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.846181 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.846703 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.846770 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.846781 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:16.847024 (+ 243us) server_negotiation.cc:299] Negotiation successful
0926 15:00:16.847076 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 15:00:16.847150 10091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.843884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:39659 (local address 127.8.102.3:46355)
0926 15:00:16.843963 (+ 79us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:16.843993 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:16.844070 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:16.844299 (+ 229us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:16.844300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:16.844302 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:16.844383 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.844385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.845068 (+ 683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.845069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.846064 (+ 995us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.846066 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.846862 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.846863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.846952 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.846956 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:00:16.847015 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 15:00:16.847065 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":59,"mutex_wait_us":27}
I20260926 15:00:16.847395 10230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.844013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:39751 (local address 127.8.102.3:46677)
0926 15:00:16.844179 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:16.844189 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:16.844246 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:16.844398 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:16.844399 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:16.844400 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:16.844484 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.844486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.845195 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.845196 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.846070 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.846072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.846813 (+ 741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.846813 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.846906 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.846911 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:00:16.847262 (+ 351us) client_negotiation.cc:240] Negotiation successful
0926 15:00:16.847317 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"mutex_wait_us":1,"thread_start_us":102,"threads_started":1}
I20260926 15:00:16.847448 10218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.843955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:46355 (local address 127.8.102.1:39659)
0926 15:00:16.844017 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:16.844019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:16.844050 (+ 31us) server_negotiation.cc:407] Connection header received
0926 15:00:16.844154 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:16.844156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:16.844170 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:16.844243 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:16.844474 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.844998 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.846173 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.846747 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.846818 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.846829 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:16.847313 (+ 484us) server_negotiation.cc:299] Negotiation successful
0926 15:00:16.847366 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 15:00:16.847734 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bdcec4a97d14b4fbc6bfad9ef14d43f" candidate_uuid: "bfe8f417978b47e69d676c497d582a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09fbced4b74c49778510ddf516a46773" is_pre_election: true
I20260926 15:00:16.847771 9872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bdcec4a97d14b4fbc6bfad9ef14d43f" candidate_uuid: "bfe8f417978b47e69d676c497d582a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3a929efe2d84366ac279ae7d3056efa" is_pre_election: true
I20260926 15:00:16.847853 10012 raft_consensus.cc:2454] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfe8f417978b47e69d676c497d582a44 in term 0.
I20260926 15:00:16.847930 9872 raft_consensus.cc:2454] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfe8f417978b47e69d676c497d582a44 in term 0.
I20260926 15:00:16.848053 10085 leader_election.cc:302] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09fbced4b74c49778510ddf516a46773, bfe8f417978b47e69d676c497d582a44; no voters:
I20260926 15:00:16.848233 10227 raft_consensus.cc:2781] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:00:16.848282 10227 raft_consensus.cc:482] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:00:16.848304 10227 raft_consensus.cc:3037] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:16.849579 10227 raft_consensus.cc:504] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } }
I20260926 15:00:16.849711 10227 leader_election.cc:288] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [CANDIDATE]: Term 1 election: Requested vote from peers b3a929efe2d84366ac279ae7d3056efa (127.8.102.1:39659), 09fbced4b74c49778510ddf516a46773 (127.8.102.2:39751)
I20260926 15:00:16.849908 9872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bdcec4a97d14b4fbc6bfad9ef14d43f" candidate_uuid: "bfe8f417978b47e69d676c497d582a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3a929efe2d84366ac279ae7d3056efa"
I20260926 15:00:16.850001 9872 raft_consensus.cc:3037] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:16.850070 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.846794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:46385 (local address 127.8.102.3:34639)
0926 15:00:16.846834 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:16.846836 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:16.847564 (+ 728us) server_negotiation.cc:407] Connection header received
0926 15:00:16.847660 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:16.847661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:16.847671 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:16.847702 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:16.847883 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.848450 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.849340 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.849689 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.849783 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.849793 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:16.849978 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 15:00:16.850017 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 15:00:16.850028 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bdcec4a97d14b4fbc6bfad9ef14d43f" candidate_uuid: "bfe8f417978b47e69d676c497d582a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09fbced4b74c49778510ddf516a46773"
I20260926 15:00:16.850107 10012 raft_consensus.cc:3037] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:16.850358 10231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.846612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:39659 (local address 127.8.102.2:50257)
0926 15:00:16.846986 (+ 374us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:16.846997 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:16.847044 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:16.847268 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:16.847269 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:16.847271 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:16.847327 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.847329 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.848182 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.848183 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.849335 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.849338 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.850112 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.850113 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.850178 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.850182 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:00:16.850215 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 15:00:16.850265 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":70,"threads_started":1}
I20260926 15:00:16.850278 10232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.846624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:34639 (local address 127.8.102.2:46385)
0926 15:00:16.847494 (+ 870us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:16.847505 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:16.847589 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:16.847757 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:16.847758 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:16.847760 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:16.847808 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.847810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.848529 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.848529 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.849243 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.849245 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.849826 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.849827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.849896 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.849900 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:00:16.850166 (+ 266us) client_negotiation.cc:240] Negotiation successful
0926 15:00:16.850216 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":853,"thread_start_us":82,"threads_started":1}
I20260926 15:00:16.850488 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.846773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:50257 (local address 127.8.102.1:39659)
0926 15:00:16.847138 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:16.847140 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:16.847147 (+ 7us) server_negotiation.cc:407] Connection header received
0926 15:00:16.847168 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:16.847168 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:16.847179 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:16.847386 (+ 207us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:16.847426 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.848088 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.849447 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.850014 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.850324 (+ 310us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.850337 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:16.850355 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 15:00:16.850395 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":44,"threads_started":1}
I20260926 15:00:16.850720 10152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bdcec4a97d14b4fbc6bfad9ef14d43f" candidate_uuid: "09fbced4b74c49778510ddf516a46773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe8f417978b47e69d676c497d582a44" is_pre_election: true
I20260926 15:00:16.850835 10152 raft_consensus.cc:2379] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09fbced4b74c49778510ddf516a46773 in current term 1: Already voted for candidate bfe8f417978b47e69d676c497d582a44 in this term.
I20260926 15:00:16.850939 9872 raft_consensus.cc:2454] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfe8f417978b47e69d676c497d582a44 in term 1.
I20260926 15:00:16.851046 10012 raft_consensus.cc:2454] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfe8f417978b47e69d676c497d582a44 in term 1.
I20260926 15:00:16.851121 10085 leader_election.cc:302] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3a929efe2d84366ac279ae7d3056efa, bfe8f417978b47e69d676c497d582a44; no voters:
I20260926 15:00:16.851141 9872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9bdcec4a97d14b4fbc6bfad9ef14d43f" candidate_uuid: "09fbced4b74c49778510ddf516a46773" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3a929efe2d84366ac279ae7d3056efa" is_pre_election: true
I20260926 15:00:16.851203 10227 raft_consensus.cc:2781] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:16.851233 9872 raft_consensus.cc:2379] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 09fbced4b74c49778510ddf516a46773 in current term 1: Already voted for candidate bfe8f417978b47e69d676c497d582a44 in this term.
I20260926 15:00:16.851380 10227 raft_consensus.cc:686] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [term 1 LEADER]: Becoming Leader. State: Replica: bfe8f417978b47e69d676c497d582a44, State: Running, Role: LEADER
I20260926 15:00:16.851483 9945 leader_election.cc:302] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 09fbced4b74c49778510ddf516a46773; no voters: b3a929efe2d84366ac279ae7d3056efa, bfe8f417978b47e69d676c497d582a44
I20260926 15:00:16.851481 10227 consensus_queue.cc:237] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } }
I20260926 15:00:16.851676 10228 raft_consensus.cc:2726] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260926 15:00:16.852365 9715 catalog_manager.cc:5495] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfe8f417978b47e69d676c497d582a44 (127.8.102.3). New cstate: current_term: 1 leader_uuid: "bfe8f417978b47e69d676c497d582a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } health_report { overall_health: UNKNOWN } } }
I20260926 15:00:16.875771 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.873177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40242 (local address 127.8.102.3:34639)
0926 15:00:16.873235 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:16.873237 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:16.873252 (+ 15us) server_negotiation.cc:407] Connection header received
0926 15:00:16.873337 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:16.873338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:16.873355 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:16.873416 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:16.873567 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.874024 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.874883 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.875080 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.875139 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.875379 (+ 240us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:16.875504 (+ 125us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:16.875605 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:16.875667 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 15:00:16.875688 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 15:00:16.877802 10012 raft_consensus.cc:1261] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773 [term 1 FOLLOWER]: Refusing update from remote peer bfe8f417978b47e69d676c497d582a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:16.877799 9872 raft_consensus.cc:1261] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa [term 1 FOLLOWER]: Refusing update from remote peer bfe8f417978b47e69d676c497d582a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:16.878216 10234 consensus_queue.cc:1035] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:16.878363 10227 consensus_queue.cc:1035] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:16.880898 10237 mvcc.cc:205] Tried to move back new op lower bound from 7333621009928146944 to 7333621009824223232. Current Snapshot: MvccSnapshot[applied={T|T < 7333621009928146944}]
I20260926 15:00:16.881206 10239 mvcc.cc:205] Tried to move back new op lower bound from 7333621009928146944 to 7333621009824223232. Current Snapshot: MvccSnapshot[applied={T|T < 7333621009928146944}]
I20260926 15:00:16.885341 9715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:16.885430 9715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:16.886327 9715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f: Sending DeleteTablet for 3 replicas of tablet 9bdcec4a97d14b4fbc6bfad9ef14d43f
I20260926 15:00:16.886804 10132 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bdcec4a97d14b4fbc6bfad9ef14d43f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:16 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:40238
I20260926 15:00:16.886830 9852 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bdcec4a97d14b4fbc6bfad9ef14d43f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:16 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:56986
I20260926 15:00:16.887082 10247 tablet_replica.cc:330] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa: stopping tablet replica
I20260926 15:00:16.887069 9992 tablet_service.cc:1514] Processing DeleteTablet for tablet 9bdcec4a97d14b4fbc6bfad9ef14d43f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:16 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:50584
I20260926 15:00:16.887269 10247 raft_consensus.cc:2229] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:16.887279 10246 tablet_replica.cc:330] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44: stopping tablet replica
I20260926 15:00:16.887359 10246 raft_consensus.cc:2229] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:00:16.887447 10247 raft_consensus.cc:2258] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:16.887408 9715 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51624:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 15:00:16.887519 10246 raft_consensus.cc:2258] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260926 15:00:16.887548 9715 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:00:16.887962 10246 ts_tablet_manager.cc:1905] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:16.887962 10247 ts_tablet_manager.cc:1905] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:16.889173 10132 tablet_service.cc:1467] Processing CreateTablet for tablet 23518d959d5145548b09af37f2906051 (DEFAULT_TABLE table=test-table [id=00de5bbd0fb4415ca75ec7813a422126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:16.889212 9992 tablet_service.cc:1467] Processing CreateTablet for tablet 23518d959d5145548b09af37f2906051 (DEFAULT_TABLE table=test-table [id=00de5bbd0fb4415ca75ec7813a422126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:16.889290 10132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23518d959d5145548b09af37f2906051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:16.889323 9992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23518d959d5145548b09af37f2906051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:16.889770 10246 ts_tablet_manager.cc:1918] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:00:16.889832 10246 log.cc:1196] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/9bdcec4a97d14b4fbc6bfad9ef14d43f
I20260926 15:00:16.890095 10246 ts_tablet_manager.cc:1939] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P bfe8f417978b47e69d676c497d582a44: Deleting consensus metadata
I20260926 15:00:16.890522 10221 tablet_bootstrap.cc:492] T 23518d959d5145548b09af37f2906051 P bfe8f417978b47e69d676c497d582a44: Bootstrap starting.
I20260926 15:00:16.890844 10222 tablet_bootstrap.cc:492] T 23518d959d5145548b09af37f2906051 P 09fbced4b74c49778510ddf516a46773: Bootstrap starting.
I20260926 15:00:16.890976 10221 tablet_bootstrap.cc:654] T 23518d959d5145548b09af37f2906051 P bfe8f417978b47e69d676c497d582a44: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:16.891361 9702 catalog_manager.cc:4841] TS bfe8f417978b47e69d676c497d582a44 (127.8.102.3:34639): tablet 9bdcec4a97d14b4fbc6bfad9ef14d43f (table test-table [id=6e0fbad878ca46c3963ba070f0fdcbb6]) successfully deleted
I20260926 15:00:16.891579 10222 tablet_bootstrap.cc:654] T 23518d959d5145548b09af37f2906051 P 09fbced4b74c49778510ddf516a46773: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:16.891819 10221 tablet_bootstrap.cc:492] T 23518d959d5145548b09af37f2906051 P bfe8f417978b47e69d676c497d582a44: No bootstrap required, opened a new log
I20260926 15:00:16.891868 10221 ts_tablet_manager.cc:1397] T 23518d959d5145548b09af37f2906051 P bfe8f417978b47e69d676c497d582a44: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:16.892002 9852 tablet_service.cc:1467] Processing CreateTablet for tablet 23518d959d5145548b09af37f2906051 (DEFAULT_TABLE table=test-table [id=00de5bbd0fb4415ca75ec7813a422126]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:16.892007 10221 raft_consensus.cc:348] T 23518d959d5145548b09af37f2906051 P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } }
I20260926 15:00:16.892073 10221 raft_consensus.cc:374] T 23518d959d5145548b09af37f2906051 P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:16.892097 10221 raft_consensus.cc:729] T 23518d959d5145548b09af37f2906051 P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe8f417978b47e69d676c497d582a44, State: Initialized, Role: FOLLOWER
I20260926 15:00:16.892102 9852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23518d959d5145548b09af37f2906051. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:16.892140 10221 consensus_queue.cc:260] T 23518d959d5145548b09af37f2906051 P bfe8f417978b47e69d676c497d582a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } }
I20260926 15:00:16.892230 10221 ts_tablet_manager.cc:1428] T 23518d959d5145548b09af37f2906051 P bfe8f417978b47e69d676c497d582a44: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:16.892949 10222 tablet_bootstrap.cc:492] T 23518d959d5145548b09af37f2906051 P 09fbced4b74c49778510ddf516a46773: No bootstrap required, opened a new log
I20260926 15:00:16.893024 10222 ts_tablet_manager.cc:1397] T 23518d959d5145548b09af37f2906051 P 09fbced4b74c49778510ddf516a46773: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 15:00:16.893180 10222 raft_consensus.cc:348] T 23518d959d5145548b09af37f2906051 P 09fbced4b74c49778510ddf516a46773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } }
I20260926 15:00:16.893254 10222 raft_consensus.cc:374] T 23518d959d5145548b09af37f2906051 P 09fbced4b74c49778510ddf516a46773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:16.893287 10222 raft_consensus.cc:729] T 23518d959d5145548b09af37f2906051 P 09fbced4b74c49778510ddf516a46773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09fbced4b74c49778510ddf516a46773, State: Initialized, Role: FOLLOWER
I20260926 15:00:16.893340 10222 consensus_queue.cc:260] T 23518d959d5145548b09af37f2906051 P 09fbced4b74c49778510ddf516a46773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } }
I20260926 15:00:16.893433 10222 ts_tablet_manager.cc:1428] T 23518d959d5145548b09af37f2906051 P 09fbced4b74c49778510ddf516a46773: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:16.894549 10224 tablet_bootstrap.cc:492] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa: Bootstrap starting.
I20260926 15:00:16.894778 10248 tablet_replica.cc:330] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773: stopping tablet replica
I20260926 15:00:16.894835 10247 ts_tablet_manager.cc:1918] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:00:16.894907 10247 log.cc:1196] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-0/wal/wals/9bdcec4a97d14b4fbc6bfad9ef14d43f
I20260926 15:00:16.895015 10224 tablet_bootstrap.cc:654] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:16.895190 10247 ts_tablet_manager.cc:1939] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P b3a929efe2d84366ac279ae7d3056efa: Deleting consensus metadata
I20260926 15:00:16.895205 10248 raft_consensus.cc:2229] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:16.895337 10248 raft_consensus.cc:2258] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:16.895798 10248 ts_tablet_manager.cc:1905] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:16.895803 9701 catalog_manager.cc:4841] TS b3a929efe2d84366ac279ae7d3056efa (127.8.102.1:39659): tablet 9bdcec4a97d14b4fbc6bfad9ef14d43f (table test-table [id=6e0fbad878ca46c3963ba070f0fdcbb6]) successfully deleted
I20260926 15:00:16.896031 10224 tablet_bootstrap.cc:492] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa: No bootstrap required, opened a new log
I20260926 15:00:16.896101 10224 ts_tablet_manager.cc:1397] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:00:16.896238 10224 raft_consensus.cc:348] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } }
I20260926 15:00:16.896287 10224 raft_consensus.cc:374] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:16.896301 10224 raft_consensus.cc:729] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3a929efe2d84366ac279ae7d3056efa, State: Initialized, Role: FOLLOWER
I20260926 15:00:16.896358 10224 consensus_queue.cc:260] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } }
I20260926 15:00:16.896456 10224 ts_tablet_manager.cc:1428] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:16.897377 10248 ts_tablet_manager.cc:1918] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:00:16.897449 10248 log.cc:1196] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/9bdcec4a97d14b4fbc6bfad9ef14d43f
I20260926 15:00:16.897722 10248 ts_tablet_manager.cc:1939] T 9bdcec4a97d14b4fbc6bfad9ef14d43f P 09fbced4b74c49778510ddf516a46773: Deleting consensus metadata
I20260926 15:00:16.898190 9701 catalog_manager.cc:4841] TS 09fbced4b74c49778510ddf516a46773 (127.8.102.2:39751): tablet 9bdcec4a97d14b4fbc6bfad9ef14d43f (table test-table [id=6e0fbad878ca46c3963ba070f0fdcbb6]) successfully deleted
I20260926 15:00:16.901629 10229 raft_consensus.cc:482] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:16.901708 10229 raft_consensus.cc:504] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } }
I20260926 15:00:16.902014 10229 leader_election.cc:288] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 09fbced4b74c49778510ddf516a46773 (127.8.102.2:39751), bfe8f417978b47e69d676c497d582a44 (127.8.102.3:34639)
I20260926 15:00:16.905366 10253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.902185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:39751 (local address 127.8.102.1:50977)
0926 15:00:16.902336 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:16.902347 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:16.902400 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:16.902773 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:16.902774 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:16.902786 (+ 12us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:16.902838 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.902840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.903671 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.903672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.904464 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.904466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.905087 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.905088 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.905168 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.905172 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:00:16.905231 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 15:00:16.905271 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"mutex_wait_us":2,"thread_start_us":87,"threads_started":1}
I20260926 15:00:16.905364 10220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.902580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:50977 (local address 127.8.102.2:39751)
0926 15:00:16.902663 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:16.902665 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:16.902673 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:00:16.902696 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:16.902697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:16.902705 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:16.902768 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:16.902905 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.903582 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.904562 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.904982 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.905053 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.905063 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:16.905245 (+ 182us) server_negotiation.cc:299] Negotiation successful
0926 15:00:16.905281 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 15:00:16.905854 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.902384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:59243 (local address 127.8.102.3:34639)
0926 15:00:16.902427 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:16.902429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:16.902437 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:00:16.902463 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:16.902463 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:16.902472 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:16.902529 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:16.902720 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.903487 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.904919 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.905437 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.905508 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.905518 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:16.905729 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 15:00:16.905769 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 15:00:16.905932 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23518d959d5145548b09af37f2906051" candidate_uuid: "b3a929efe2d84366ac279ae7d3056efa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09fbced4b74c49778510ddf516a46773" is_pre_election: true
I20260926 15:00:16.906038 10012 raft_consensus.cc:2454] T 23518d959d5145548b09af37f2906051 P 09fbced4b74c49778510ddf516a46773 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3a929efe2d84366ac279ae7d3056efa in term 0.
I20260926 15:00:16.906147 10252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.902185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:34639 (local address 127.8.102.1:59243)
0926 15:00:16.902322 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:16.902334 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:16.902394 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:16.902526 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:16.902527 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:16.902529 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:16.902600 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.902601 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.903578 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.903579 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.904828 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.904831 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.905552 (+ 721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.905553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.905642 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.905647 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:00:16.906013 (+ 366us) client_negotiation.cc:240] Negotiation successful
0926 15:00:16.906068 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260926 15:00:16.906201 9805 leader_election.cc:302] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09fbced4b74c49778510ddf516a46773, b3a929efe2d84366ac279ae7d3056efa; no voters:
I20260926 15:00:16.906335 10229 raft_consensus.cc:2781] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:00:16.906381 10229 raft_consensus.cc:482] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:00:16.906404 10229 raft_consensus.cc:3037] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:16.906445 10152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23518d959d5145548b09af37f2906051" candidate_uuid: "b3a929efe2d84366ac279ae7d3056efa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe8f417978b47e69d676c497d582a44" is_pre_election: true
I20260926 15:00:16.906524 10152 raft_consensus.cc:2454] T 23518d959d5145548b09af37f2906051 P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b3a929efe2d84366ac279ae7d3056efa in term 0.
I20260926 15:00:16.907389 10229 raft_consensus.cc:504] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } }
I20260926 15:00:16.907526 10229 leader_election.cc:288] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [CANDIDATE]: Term 1 election: Requested vote from peers 09fbced4b74c49778510ddf516a46773 (127.8.102.2:39751), bfe8f417978b47e69d676c497d582a44 (127.8.102.3:34639)
I20260926 15:00:16.907691 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23518d959d5145548b09af37f2906051" candidate_uuid: "b3a929efe2d84366ac279ae7d3056efa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09fbced4b74c49778510ddf516a46773"
I20260926 15:00:16.907787 10012 raft_consensus.cc:3037] T 23518d959d5145548b09af37f2906051 P 09fbced4b74c49778510ddf516a46773 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:16.908501 10012 raft_consensus.cc:2454] T 23518d959d5145548b09af37f2906051 P 09fbced4b74c49778510ddf516a46773 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3a929efe2d84366ac279ae7d3056efa in term 1.
I20260926 15:00:16.908622 10152 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23518d959d5145548b09af37f2906051" candidate_uuid: "b3a929efe2d84366ac279ae7d3056efa" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bfe8f417978b47e69d676c497d582a44"
I20260926 15:00:16.908699 9805 leader_election.cc:302] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09fbced4b74c49778510ddf516a46773, b3a929efe2d84366ac279ae7d3056efa; no voters:
I20260926 15:00:16.908716 10152 raft_consensus.cc:3037] T 23518d959d5145548b09af37f2906051 P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:16.908780 10229 raft_consensus.cc:2781] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:16.908929 10229 raft_consensus.cc:686] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [term 1 LEADER]: Becoming Leader. State: Replica: b3a929efe2d84366ac279ae7d3056efa, State: Running, Role: LEADER
I20260926 15:00:16.908989 10229 consensus_queue.cc:237] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } }
I20260926 15:00:16.909603 10152 raft_consensus.cc:2454] T 23518d959d5145548b09af37f2906051 P bfe8f417978b47e69d676c497d582a44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b3a929efe2d84366ac279ae7d3056efa in term 1.
I20260926 15:00:16.909653 9715 catalog_manager.cc:5495] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa reported cstate change: term changed from 0 to 1, leader changed from <none> to b3a929efe2d84366ac279ae7d3056efa (127.8.102.1). New cstate: current_term: 1 leader_uuid: "b3a929efe2d84366ac279ae7d3056efa" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } health_report { overall_health: UNKNOWN } } }
Sep 26 15:00:16 dist-test-slave-jnhs krb5kdc[9670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434815, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:16.943782 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.935582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36745 (local address 127.0.0.1:51626)
0926 15:00:16.935763 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:16.935775 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:16.935847 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:16.936000 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:16.936001 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:16.936462 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:16.936562 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.936565 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.937512 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.937513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.938083 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:16.938086 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.938571 (+ 485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.938572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:16.938616 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.939308 (+ 692us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:16.939313 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:16.940796 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:16.942572 (+ 1776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:16.942580 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:16.942582 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:16.942867 (+ 285us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:16.943083 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:16.943083 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:16.943084 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:16.943149 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:16.943474 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:16.943482 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:16.943531 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 15:00:16.943600 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 15:00:16.943705 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":118,"threads_started":1}
I20260926 15:00:16.943881 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:16.935617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51626 (local address 127.8.102.62:36745)
0926 15:00:16.935667 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:16.935670 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:16.935851 (+ 181us) server_negotiation.cc:407] Connection header received
0926 15:00:16.935914 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:16.935915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:16.935927 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:16.935986 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:16.936655 (+ 669us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.937422 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.938172 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:16.938475 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:16.938548 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:16.940893 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:16.940896 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:16.940896 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:16.940904 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:16.942480 (+ 1576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:16.942942 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:16.942943 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:16.942943 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:16.943007 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:16.943210 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:16.943211 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:16.943211 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:16.943353 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:16.943423 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:16.943645 (+ 222us) server_negotiation.cc:299] Negotiation successful
0926 15:00:16.943768 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":23}
I20260926 15:00:16.946288 9715 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:51626:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 15:00:16.946557 9715 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:00:16.948184 10132 tablet_service.cc:1467] Processing CreateTablet for tablet 5a686d108a6b4709aa273585a9d42257 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50fa8d86e7e447adb2c849cf653471a3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:00:16.948213 9852 tablet_service.cc:1467] Processing CreateTablet for tablet 5a686d108a6b4709aa273585a9d42257 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50fa8d86e7e447adb2c849cf653471a3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:00:16.948246 9992 tablet_service.cc:1467] Processing CreateTablet for tablet 5a686d108a6b4709aa273585a9d42257 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=50fa8d86e7e447adb2c849cf653471a3]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:00:16.948359 10132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a686d108a6b4709aa273585a9d42257. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:16.948365 9852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a686d108a6b4709aa273585a9d42257. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:16.948386 9992 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5a686d108a6b4709aa273585a9d42257. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:16.949831 10224 tablet_bootstrap.cc:492] T 5a686d108a6b4709aa273585a9d42257 P b3a929efe2d84366ac279ae7d3056efa: Bootstrap starting.
I20260926 15:00:16.949839 10221 tablet_bootstrap.cc:492] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44: Bootstrap starting.
I20260926 15:00:16.950212 10221 tablet_bootstrap.cc:654] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:16.950222 10224 tablet_bootstrap.cc:654] T 5a686d108a6b4709aa273585a9d42257 P b3a929efe2d84366ac279ae7d3056efa: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:16.950711 10222 tablet_bootstrap.cc:492] T 5a686d108a6b4709aa273585a9d42257 P 09fbced4b74c49778510ddf516a46773: Bootstrap starting.
I20260926 15:00:16.951028 10221 tablet_bootstrap.cc:492] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44: No bootstrap required, opened a new log
I20260926 15:00:16.951028 10224 tablet_bootstrap.cc:492] T 5a686d108a6b4709aa273585a9d42257 P b3a929efe2d84366ac279ae7d3056efa: No bootstrap required, opened a new log
I20260926 15:00:16.951072 10224 ts_tablet_manager.cc:1397] T 5a686d108a6b4709aa273585a9d42257 P b3a929efe2d84366ac279ae7d3056efa: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:16.951073 10221 ts_tablet_manager.cc:1397] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:16.951139 10222 tablet_bootstrap.cc:654] T 5a686d108a6b4709aa273585a9d42257 P 09fbced4b74c49778510ddf516a46773: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:16.951223 10221 raft_consensus.cc:348] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } }
I20260926 15:00:16.951215 10224 raft_consensus.cc:348] T 5a686d108a6b4709aa273585a9d42257 P b3a929efe2d84366ac279ae7d3056efa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } }
I20260926 15:00:16.951287 10221 raft_consensus.cc:374] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:16.951287 10224 raft_consensus.cc:374] T 5a686d108a6b4709aa273585a9d42257 P b3a929efe2d84366ac279ae7d3056efa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:16.951308 10224 raft_consensus.cc:729] T 5a686d108a6b4709aa273585a9d42257 P b3a929efe2d84366ac279ae7d3056efa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b3a929efe2d84366ac279ae7d3056efa, State: Initialized, Role: FOLLOWER
I20260926 15:00:16.951308 10221 raft_consensus.cc:729] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bfe8f417978b47e69d676c497d582a44, State: Initialized, Role: FOLLOWER
I20260926 15:00:16.951341 10221 consensus_queue.cc:260] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } }
I20260926 15:00:16.951354 10224 consensus_queue.cc:260] T 5a686d108a6b4709aa273585a9d42257 P b3a929efe2d84366ac279ae7d3056efa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } }
I20260926 15:00:16.951438 10224 ts_tablet_manager.cc:1428] T 5a686d108a6b4709aa273585a9d42257 P b3a929efe2d84366ac279ae7d3056efa: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:16.951447 10221 ts_tablet_manager.cc:1428] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:16.951574 10229 tablet_replica.cc:439] T 5a686d108a6b4709aa273585a9d42257 P b3a929efe2d84366ac279ae7d3056efa: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } } }
I20260926 15:00:16.951658 10229 tablet_replica.cc:442] T 5a686d108a6b4709aa273585a9d42257 P b3a929efe2d84366ac279ae7d3056efa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:00:16.951745 10227 tablet_replica.cc:439] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } } }
I20260926 15:00:16.951835 10227 tablet_replica.cc:442] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:00:16.954097 10222 tablet_bootstrap.cc:492] T 5a686d108a6b4709aa273585a9d42257 P 09fbced4b74c49778510ddf516a46773: No bootstrap required, opened a new log
I20260926 15:00:16.954156 10222 ts_tablet_manager.cc:1397] T 5a686d108a6b4709aa273585a9d42257 P 09fbced4b74c49778510ddf516a46773: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260926 15:00:16.954334 10222 raft_consensus.cc:348] T 5a686d108a6b4709aa273585a9d42257 P 09fbced4b74c49778510ddf516a46773 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } }
I20260926 15:00:16.954393 10222 raft_consensus.cc:374] T 5a686d108a6b4709aa273585a9d42257 P 09fbced4b74c49778510ddf516a46773 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:16.954411 10222 raft_consensus.cc:729] T 5a686d108a6b4709aa273585a9d42257 P 09fbced4b74c49778510ddf516a46773 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09fbced4b74c49778510ddf516a46773, State: Initialized, Role: FOLLOWER
I20260926 15:00:16.954468 10222 consensus_queue.cc:260] T 5a686d108a6b4709aa273585a9d42257 P 09fbced4b74c49778510ddf516a46773 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } }
I20260926 15:00:16.954558 10222 ts_tablet_manager.cc:1428] T 5a686d108a6b4709aa273585a9d42257 P 09fbced4b74c49778510ddf516a46773: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:16.954619 10228 tablet_replica.cc:439] T 5a686d108a6b4709aa273585a9d42257 P 09fbced4b74c49778510ddf516a46773: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } } }
I20260926 15:00:16.954751 10228 tablet_replica.cc:442] T 5a686d108a6b4709aa273585a9d42257 P 09fbced4b74c49778510ddf516a46773: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:00:17.146104 10227 raft_consensus.cc:482] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:17.146215 10227 raft_consensus.cc:504] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } }
I20260926 15:00:17.146420 10227 leader_election.cc:288] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b3a929efe2d84366ac279ae7d3056efa (127.8.102.1:39659), 09fbced4b74c49778510ddf516a46773 (127.8.102.2:39751)
I20260926 15:00:17.146724 9872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a686d108a6b4709aa273585a9d42257" candidate_uuid: "bfe8f417978b47e69d676c497d582a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3a929efe2d84366ac279ae7d3056efa" is_pre_election: true
I20260926 15:00:17.146733 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a686d108a6b4709aa273585a9d42257" candidate_uuid: "bfe8f417978b47e69d676c497d582a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09fbced4b74c49778510ddf516a46773" is_pre_election: true
I20260926 15:00:17.146826 10012 raft_consensus.cc:2454] T 5a686d108a6b4709aa273585a9d42257 P 09fbced4b74c49778510ddf516a46773 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfe8f417978b47e69d676c497d582a44 in term 0.
I20260926 15:00:17.146826 9872 raft_consensus.cc:2454] T 5a686d108a6b4709aa273585a9d42257 P b3a929efe2d84366ac279ae7d3056efa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bfe8f417978b47e69d676c497d582a44 in term 0.
I20260926 15:00:17.146957 10085 leader_election.cc:302] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b3a929efe2d84366ac279ae7d3056efa, bfe8f417978b47e69d676c497d582a44; no voters:
I20260926 15:00:17.147105 10227 raft_consensus.cc:2781] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:00:17.147156 10227 raft_consensus.cc:482] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:00:17.147187 10227 raft_consensus.cc:3037] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:17.147893 10227 raft_consensus.cc:504] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } }
I20260926 15:00:17.148036 10227 leader_election.cc:288] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44 [CANDIDATE]: Term 1 election: Requested vote from peers b3a929efe2d84366ac279ae7d3056efa (127.8.102.1:39659), 09fbced4b74c49778510ddf516a46773 (127.8.102.2:39751)
I20260926 15:00:17.148253 9872 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a686d108a6b4709aa273585a9d42257" candidate_uuid: "bfe8f417978b47e69d676c497d582a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b3a929efe2d84366ac279ae7d3056efa"
I20260926 15:00:17.148324 9872 raft_consensus.cc:3037] T 5a686d108a6b4709aa273585a9d42257 P b3a929efe2d84366ac279ae7d3056efa [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:17.148311 10012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5a686d108a6b4709aa273585a9d42257" candidate_uuid: "bfe8f417978b47e69d676c497d582a44" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "09fbced4b74c49778510ddf516a46773"
I20260926 15:00:17.148425 10012 raft_consensus.cc:3037] T 5a686d108a6b4709aa273585a9d42257 P 09fbced4b74c49778510ddf516a46773 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:17.149015 9872 raft_consensus.cc:2454] T 5a686d108a6b4709aa273585a9d42257 P b3a929efe2d84366ac279ae7d3056efa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfe8f417978b47e69d676c497d582a44 in term 1.
I20260926 15:00:17.149027 10012 raft_consensus.cc:2454] T 5a686d108a6b4709aa273585a9d42257 P 09fbced4b74c49778510ddf516a46773 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bfe8f417978b47e69d676c497d582a44 in term 1.
I20260926 15:00:17.149235 10085 leader_election.cc:302] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 09fbced4b74c49778510ddf516a46773, bfe8f417978b47e69d676c497d582a44; no voters:
I20260926 15:00:17.149322 10227 raft_consensus.cc:2781] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:17.149374 10227 raft_consensus.cc:686] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44 [term 1 LEADER]: Becoming Leader. State: Replica: bfe8f417978b47e69d676c497d582a44, State: Running, Role: LEADER
I20260926 15:00:17.149426 10227 consensus_queue.cc:237] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } }
I20260926 15:00:17.149709 10234 tablet_replica.cc:439] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44: TxnStatusTablet state changed. Reason: New leader bfe8f417978b47e69d676c497d582a44. Latest consensus state: current_term: 1 leader_uuid: "bfe8f417978b47e69d676c497d582a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } } }
I20260926 15:00:17.149806 10234 tablet_replica.cc:442] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:00:17.149972 10266 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 15:00:17.150087 10266 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 15:00:17.150130 9715 catalog_manager.cc:5495] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44 reported cstate change: term changed from 0 to 1, leader changed from <none> to bfe8f417978b47e69d676c497d582a44 (127.8.102.3). New cstate: current_term: 1 leader_uuid: "bfe8f417978b47e69d676c497d582a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } health_report { overall_health: UNKNOWN } } }
I20260926 15:00:17.185470 10261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:17.182976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:34639 (local address 127.0.0.1:40244)
0926 15:00:17.183058 (+ 82us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:17.183086 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:17.183148 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:17.183303 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:17.183304 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:17.183305 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:00:17.183358 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:17.183360 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:17.184013 (+ 653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:17.184015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:17.184598 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:17.184600 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:17.184983 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:17.184984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:17.185057 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:17.185072 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:00:17.185279 (+ 207us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:00:17.185288 (+ 9us) client_negotiation.cc:769] Sending connection context
0926 15:00:17.185353 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 15:00:17.185402 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 15:00:17.185457 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:17.183049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.8.102.3:34639)
0926 15:00:17.183112 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:17.183113 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:17.183124 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:00:17.183233 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:17.183234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:17.183246 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:17.183297 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:17.183431 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:17.183919 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:17.184691 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:17.184905 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:17.184982 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:17.185170 (+ 188us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:17.185206 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:17.185276 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:17.185358 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 15:00:17.185375 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 15:00:17.186563 9872 raft_consensus.cc:1261] T 5a686d108a6b4709aa273585a9d42257 P b3a929efe2d84366ac279ae7d3056efa [term 1 FOLLOWER]: Refusing update from remote peer bfe8f417978b47e69d676c497d582a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:17.186595 10012 raft_consensus.cc:1261] T 5a686d108a6b4709aa273585a9d42257 P 09fbced4b74c49778510ddf516a46773 [term 1 FOLLOWER]: Refusing update from remote peer bfe8f417978b47e69d676c497d582a44: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:17.186833 10227 consensus_queue.cc:1035] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:17.186992 10227 consensus_queue.cc:1035] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44 [LEADER]: Connected to new peer: Peer: permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:17.187492 10228 tablet_replica.cc:439] T 5a686d108a6b4709aa273585a9d42257 P 09fbced4b74c49778510ddf516a46773: TxnStatusTablet state changed. Reason: New leader bfe8f417978b47e69d676c497d582a44. Latest consensus state: current_term: 1 leader_uuid: "bfe8f417978b47e69d676c497d582a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } } }
I20260926 15:00:17.187601 10228 tablet_replica.cc:442] T 5a686d108a6b4709aa273585a9d42257 P 09fbced4b74c49778510ddf516a46773: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:00:17.188488 10227 tablet_replica.cc:439] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bfe8f417978b47e69d676c497d582a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } } }
I20260926 15:00:17.188580 10227 tablet_replica.cc:442] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:00:17.188827 10229 tablet_replica.cc:439] T 5a686d108a6b4709aa273585a9d42257 P b3a929efe2d84366ac279ae7d3056efa: TxnStatusTablet state changed. Reason: New leader bfe8f417978b47e69d676c497d582a44. Latest consensus state: current_term: 1 leader_uuid: "bfe8f417978b47e69d676c497d582a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } } }
I20260926 15:00:17.188915 10229 tablet_replica.cc:442] T 5a686d108a6b4709aa273585a9d42257 P b3a929efe2d84366ac279ae7d3056efa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:00:17.189220 10234 tablet_replica.cc:439] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bfe8f417978b47e69d676c497d582a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } } }
I20260926 15:00:17.189304 10234 tablet_replica.cc:442] T 5a686d108a6b4709aa273585a9d42257 P bfe8f417978b47e69d676c497d582a44: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:00:17.189322 10238 mvcc.cc:205] Tried to move back new op lower bound from 7333621011194474496 to 7333621011044384768. Current Snapshot: MvccSnapshot[applied={T|T < 7333621011194474496}]
I20260926 15:00:17.189438 10228 tablet_replica.cc:439] T 5a686d108a6b4709aa273585a9d42257 P 09fbced4b74c49778510ddf516a46773: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bfe8f417978b47e69d676c497d582a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } } }
I20260926 15:00:17.189512 10228 tablet_replica.cc:442] T 5a686d108a6b4709aa273585a9d42257 P 09fbced4b74c49778510ddf516a46773: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:00:17.190186 10229 tablet_replica.cc:439] T 5a686d108a6b4709aa273585a9d42257 P b3a929efe2d84366ac279ae7d3056efa: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bfe8f417978b47e69d676c497d582a44" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b3a929efe2d84366ac279ae7d3056efa" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 39659 } } peers { permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 } } peers { permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 } } }
I20260926 15:00:17.190264 10229 tablet_replica.cc:442] T 5a686d108a6b4709aa273585a9d42257 P b3a929efe2d84366ac279ae7d3056efa: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:00:17.193359 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:17.190669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.8.102.1:39659)
0926 15:00:17.190709 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:17.190711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:17.190729 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:00:17.190762 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:17.190762 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:17.190781 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:17.190844 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:17.191005 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:17.191740 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:17.192482 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:17.192739 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:17.192810 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:17.192994 (+ 184us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:17.193134 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:17.193208 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:17.193264 (+ 56us) server_negotiation.cc:299] Negotiation successful
0926 15:00:17.193288 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 15:00:17.198405 10217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:17.195907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:43593 (local address 127.8.102.3:34639)
0926 15:00:17.195954 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:17.195956 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:17.196050 (+ 94us) server_negotiation.cc:407] Connection header received
0926 15:00:17.196132 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:17.196133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:17.196143 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:17.196201 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:17.196353 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:17.196913 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:17.197686 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:17.197883 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:17.197958 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:17.198148 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:17.198186 (+ 38us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:17.198246 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:17.198307 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 15:00:17.198326 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 15:00:17.198490 10273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:17.195841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:34639 (local address 127.8.102.1:43593)
0926 15:00:17.195985 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:17.195995 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:17.196065 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:17.196206 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:17.196207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:17.196208 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:00:17.196278 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:17.196280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:17.197017 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:17.197018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:17.197593 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:17.197594 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:17.197969 (+ 375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:17.197970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:17.198041 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:17.198056 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:00:17.198247 (+ 191us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:00:17.198250 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:00:17.198303 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 15:00:17.198351 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
I20260926 15:00:17.200879 10012 raft_consensus.cc:1261] T 23518d959d5145548b09af37f2906051 P 09fbced4b74c49778510ddf516a46773 [term 1 FOLLOWER]: Refusing update from remote peer b3a929efe2d84366ac279ae7d3056efa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:17.200999 10152 raft_consensus.cc:1261] T 23518d959d5145548b09af37f2906051 P bfe8f417978b47e69d676c497d582a44 [term 1 FOLLOWER]: Refusing update from remote peer b3a929efe2d84366ac279ae7d3056efa: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:17.201164 10254 consensus_queue.cc:1035] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [LEADER]: Connected to new peer: Peer: permanent_uuid: "09fbced4b74c49778510ddf516a46773" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 39751 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:17.201313 10254 consensus_queue.cc:1035] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [LEADER]: Connected to new peer: Peer: permanent_uuid: "bfe8f417978b47e69d676c497d582a44" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:17.210500 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:17.207597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:35583 (local address 127.8.102.1:39659)
0926 15:00:17.207637 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:17.207639 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:17.207844 (+ 205us) server_negotiation.cc:407] Connection header received
0926 15:00:17.207923 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:17.207924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:17.207932 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:17.207976 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:17.208094 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:17.208549 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:17.209783 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:17.210048 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:17.210107 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:17.210297 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:17.210331 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:17.210369 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:17.210421 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 15:00:17.210440 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 15:00:17.210544 10278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:17.207448 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:39659 (local address 127.8.102.3:35583)
0926 15:00:17.207783 (+ 335us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:17.207794 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:17.207868 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:17.207979 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:17.207980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:17.207981 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:00:17.208042 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:17.208043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:17.208633 (+ 590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:17.208634 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:17.209653 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:17.209656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:17.210126 (+ 470us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:17.210127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:17.210190 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:17.210213 (+ 23us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:00:17.210370 (+ 157us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:00:17.210372 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 15:00:17.210418 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 15:00:17.210465 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":66,"threads_started":1}
I20260926 15:00:17.219937 9715 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:17.220080 9715 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51624:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:17.220749 9715 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1ff0f95a826245dcae03caf1969e076f: Sending DeleteTablet for 3 replicas of tablet 23518d959d5145548b09af37f2906051
I20260926 15:00:17.221058 10132 tablet_service.cc:1514] Processing DeleteTablet for tablet 23518d959d5145548b09af37f2906051 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:17 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:40238
I20260926 15:00:17.221117 9852 tablet_service.cc:1514] Processing DeleteTablet for tablet 23518d959d5145548b09af37f2906051 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:17 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:56986
I20260926 15:00:17.221210 10247 tablet_replica.cc:330] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa: stopping tablet replica
I20260926 15:00:17.221257 10247 raft_consensus.cc:2229] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:00:17.221342 10247 raft_consensus.cc:2258] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:17.221716 10246 tablet_replica.cc:330] T 23518d959d5145548b09af37f2906051 P bfe8f417978b47e69d676c497d582a44: stopping tablet replica
I20260926 15:00:17.221819 10246 raft_consensus.cc:2229] T 23518d959d5145548b09af37f2906051 P bfe8f417978b47e69d676c497d582a44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:17.221864 10246 raft_consensus.cc:2258] T 23518d959d5145548b09af37f2906051 P bfe8f417978b47e69d676c497d582a44 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:17.222175 10246 ts_tablet_manager.cc:1905] T 23518d959d5145548b09af37f2906051 P bfe8f417978b47e69d676c497d582a44: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:17.222177 10247 ts_tablet_manager.cc:1905] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:17.223968 9992 tablet_service.cc:1514] Processing DeleteTablet for tablet 23518d959d5145548b09af37f2906051 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:17 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:50584
I20260926 15:00:17.224116 10248 tablet_replica.cc:330] T 23518d959d5145548b09af37f2906051 P 09fbced4b74c49778510ddf516a46773: stopping tablet replica
I20260926 15:00:17.224174 10246 ts_tablet_manager.cc:1918] T 23518d959d5145548b09af37f2906051 P bfe8f417978b47e69d676c497d582a44: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:00:17.224190 10248 raft_consensus.cc:2229] T 23518d959d5145548b09af37f2906051 P 09fbced4b74c49778510ddf516a46773 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:17.224232 10248 raft_consensus.cc:2258] T 23518d959d5145548b09af37f2906051 P 09fbced4b74c49778510ddf516a46773 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:17.224232 10246 log.cc:1196] T 23518d959d5145548b09af37f2906051 P bfe8f417978b47e69d676c497d582a44: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/23518d959d5145548b09af37f2906051
I20260926 15:00:17.224270 10247 ts_tablet_manager.cc:1918] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:00:17.224336 10247 log.cc:1196] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-0/wal/wals/23518d959d5145548b09af37f2906051
I20260926 15:00:17.224483 10246 ts_tablet_manager.cc:1939] T 23518d959d5145548b09af37f2906051 P bfe8f417978b47e69d676c497d582a44: Deleting consensus metadata
I20260926 15:00:17.224557 10247 ts_tablet_manager.cc:1939] T 23518d959d5145548b09af37f2906051 P b3a929efe2d84366ac279ae7d3056efa: Deleting consensus metadata
I20260926 15:00:17.224607 10248 ts_tablet_manager.cc:1905] T 23518d959d5145548b09af37f2906051 P 09fbced4b74c49778510ddf516a46773: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:17.224865 9702 catalog_manager.cc:4841] TS bfe8f417978b47e69d676c497d582a44 (127.8.102.3:34639): tablet 23518d959d5145548b09af37f2906051 (table test-table [id=00de5bbd0fb4415ca75ec7813a422126]) successfully deleted
I20260926 15:00:17.224865 9701 catalog_manager.cc:4841] TS b3a929efe2d84366ac279ae7d3056efa (127.8.102.1:39659): tablet 23518d959d5145548b09af37f2906051 (table test-table [id=00de5bbd0fb4415ca75ec7813a422126]) successfully deleted
I20260926 15:00:17.226312 10248 ts_tablet_manager.cc:1918] T 23518d959d5145548b09af37f2906051 P 09fbced4b74c49778510ddf516a46773: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:00:17.226378 10248 log.cc:1196] T 23518d959d5145548b09af37f2906051 P 09fbced4b74c49778510ddf516a46773: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/23518d959d5145548b09af37f2906051
I20260926 15:00:17.226624 10248 ts_tablet_manager.cc:1939] T 23518d959d5145548b09af37f2906051 P 09fbced4b74c49778510ddf516a46773: Deleting consensus metadata
I20260926 15:00:17.227211 9701 catalog_manager.cc:4841] TS 09fbced4b74c49778510ddf516a46773 (127.8.102.2:39751): tablet 23518d959d5145548b09af37f2906051 (table test-table [id=00de5bbd0fb4415ca75ec7813a422126]) successfully deleted
Sep 26 15:00:17 dist-test-slave-jnhs krb5kdc[9670](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434816, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.102.1@KRBTEST.COM
I20260926 15:00:17.243875 10233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:17.231034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56990 (local address 127.8.102.1:39659)
0926 15:00:17.231089 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:17.231091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:17.231260 (+ 169us) server_negotiation.cc:407] Connection header received
0926 15:00:17.231316 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:17.231317 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:17.231330 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:17.231406 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:17.232116 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:17.232870 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:17.233990 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:17.234295 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:17.234369 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:17.240239 (+ 5870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:17.240248 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:17.240249 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:17.240258 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:17.242413 (+ 2155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:17.242910 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:17.242914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:17.242915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:17.242985 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:17.243217 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:17.243217 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:17.243218 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:17.243409 (+ 191us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:17.243577 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:17.243673 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 15:00:17.243761 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260926 15:00:17.244240 9912 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:56990
I20260926 15:00:17.253134 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:17.245223 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51632 (local address 127.8.102.62:36745)
0926 15:00:17.245271 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:17.245272 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:17.245467 (+ 195us) server_negotiation.cc:407] Connection header received
0926 15:00:17.245598 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:17.245599 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:17.245610 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:17.245641 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:17.246325 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:17.247187 (+ 862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:17.248231 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:17.248548 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:17.248590 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:17.250142 (+ 1552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:17.250143 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:17.250143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:17.250150 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:17.251881 (+ 1731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:17.252271 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:17.252272 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:17.252273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:17.252330 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:17.252525 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:17.252525 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:17.252526 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:17.252667 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:17.252748 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:17.252953 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 15:00:17.253033 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
W20260926 15:00:17.253911 9715 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51632
I20260926 15:00:17.254894 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 9787
I20260926 15:00:17.263851 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 9927
I20260926 15:00:17.269064 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 10067
I20260926 15:00:17.274793 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 9683
2026-09-26T15:00:17Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2623 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:00:17 dist-test-slave-jnhs krb5kdc[10287](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:00:17 dist-test-slave-jnhs krb5kdc[10287](info): set up 2 sockets
Sep 26 15:00:17 dist-test-slave-jnhs krb5kdc[10287](info): commencing operation
krb5kdc: starting...
W20260926 15:00:18.342526 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:00:18 dist-test-slave-jnhs krb5kdc[10287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:00:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:00:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:00:18.470197 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:37383
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:37383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:18.551323 10300 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:18.551637 10300 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:18.551679 10300 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:18.552901 10300 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:18.552974 10300 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 15:00:18.552996 10300 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:18.553041 10300 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:18.553088 10300 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:18.553117 10300 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:00:18.554782 10300 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:37383
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:37383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10300
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:18.555066 10300 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:18.555305 10300 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:18.560314 10308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:18.560297 10306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:18.560283 10305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:18.560370 10300 server_base.cc:1034] running on GCE node
I20260926 15:00:18.560827 10300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:18.561174 10300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:18.562362 10300 hybrid_clock.cc:648] HybridClock initialized: now 1790434818562338 us; error 45 us; skew 500 ppm
Sep 26 15:00:18 dist-test-slave-jnhs krb5kdc[10287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434818, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:18.565219 10300 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:00:18.565690 10300 webserver.cc:460] Webserver started at http://127.8.102.62:37859/ using document root <none> and password file <none>
I20260926 15:00:18.565893 10300 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:18.565937 10300 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:18.566041 10300 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:18.567296 10300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "8050e2da320f4e5c95099067a5064b78"
format_stamp: "Formatted at 2026-09-26 15:00:18 on dist-test-slave-jnhs"
server_key: "6926860be09108a501a6a7a149065f31"
server_key_iv: "e35838217468516aede77f8300c1441b"
server_key_version: "encryptionkey@0"
I20260926 15:00:18.567659 10300 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "8050e2da320f4e5c95099067a5064b78"
format_stamp: "Formatted at 2026-09-26 15:00:18 on dist-test-slave-jnhs"
server_key: "6926860be09108a501a6a7a149065f31"
server_key_iv: "e35838217468516aede77f8300c1441b"
server_key_version: "encryptionkey@0"
I20260926 15:00:18.569166 10300 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:00:18.570083 10315 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:18.570322 10300 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:18.570427 10300 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "8050e2da320f4e5c95099067a5064b78"
format_stamp: "Formatted at 2026-09-26 15:00:18 on dist-test-slave-jnhs"
server_key: "6926860be09108a501a6a7a149065f31"
server_key_iv: "e35838217468516aede77f8300c1441b"
server_key_version: "encryptionkey@0"
I20260926 15:00:18.570497 10300 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:18.583730 10300 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:18.586073 10300 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:18.586192 10300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:18.590850 10300 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:37383
I20260926 15:00:18.590986 10377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:37383 every 8 connection(s)
I20260926 15:00:18.591289 10300 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:00:18.592137 10378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:18.594620 10378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78: Bootstrap starting.
I20260926 15:00:18.595237 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 10300
I20260926 15:00:18.595314 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:00:18.595397 10378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:18.595502 8600 external_mini_cluster.cc:1428] Setting key 430cac21cabb228f2b8c8d8b632c751b
I20260926 15:00:18.595834 10378 log.cc:824] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:18.596692 10378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78: No bootstrap required, opened a new log
I20260926 15:00:18.598510 10378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8050e2da320f4e5c95099067a5064b78" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37383 } }
I20260926 15:00:18.598706 10378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:18.598773 10378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8050e2da320f4e5c95099067a5064b78, State: Initialized, Role: FOLLOWER
I20260926 15:00:18.598903 10378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8050e2da320f4e5c95099067a5064b78" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37383 } }
I20260926 15:00:18.598966 10378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:00:18.598987 10378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:00:18.599056 10378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:18.599639 10378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8050e2da320f4e5c95099067a5064b78" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37383 } }
I20260926 15:00:18.599748 10378 leader_election.cc:302] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8050e2da320f4e5c95099067a5064b78; no voters:
I20260926 15:00:18.599907 10378 leader_election.cc:288] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:00:18.599989 10383 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:18.600158 10378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:18.600198 10383 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78 [term 1 LEADER]: Becoming Leader. State: Replica: 8050e2da320f4e5c95099067a5064b78, State: Running, Role: LEADER
I20260926 15:00:18.600402 10383 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8050e2da320f4e5c95099067a5064b78" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37383 } }
I20260926 15:00:18.600786 10384 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8050e2da320f4e5c95099067a5064b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8050e2da320f4e5c95099067a5064b78" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37383 } } }
I20260926 15:00:18.600864 10384 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:18.601141 10385 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8050e2da320f4e5c95099067a5064b78. Latest consensus state: current_term: 1 leader_uuid: "8050e2da320f4e5c95099067a5064b78" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8050e2da320f4e5c95099067a5064b78" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37383 } } }
I20260926 15:00:18.601495 10385 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:18.601687 10392 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:00:18.602579 10392 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 15:00:18 dist-test-slave-jnhs krb5kdc[10287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:18.604315 10392 catalog_manager.cc:1347] Generated new cluster ID: b9e7ae216582495a9429fafcd559edc3
I20260926 15:00:18.604368 10392 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:00:18.608860 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:18.596096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51500 (local address 127.8.102.62:37383)
0926 15:00:18.596282 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:18.596293 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:18.596333 (+ 40us) server_negotiation.cc:407] Connection header received
0926 15:00:18.596938 (+ 605us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:18.596946 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:18.597009 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:18.597138 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:18.597892 (+ 754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:18.599516 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:18.600586 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:18.601296 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:18.601423 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:18.604610 (+ 3187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:18.604614 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:18.604614 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:18.604629 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:18.606707 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:18.607141 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:18.607144 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:18.607145 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:18.607248 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:18.607468 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:18.607469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:18.607469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:18.607787 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:18.607939 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:18.608401 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 15:00:18.608563 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260926 15:00:18.615778 10392 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:00:18.616356 10392 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:00:18.621860 10392 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78: Generated new TSK 0
I20260926 15:00:18.622100 10392 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
I20260926 15:00:18.653874 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37383
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:18.740805 10404 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:18.741057 10404 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:18.741112 10404 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:18.742394 10404 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:18.742441 10404 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:18.742463 10404 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 15:00:18.742475 10404 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:00:18.742492 10404 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:00:18.744065 10404 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.102.62:37383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10404
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:18.744323 10404 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:18.744581 10404 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:18.749795 10409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:18.749786 10410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:18.749818 10412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:18.749858 10404 server_base.cc:1034] running on GCE node
I20260926 15:00:18.750144 10404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:18.750350 10404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:18.751490 10404 hybrid_clock.cc:648] HybridClock initialized: now 1790434818751465 us; error 41 us; skew 500 ppm
Sep 26 15:00:18 dist-test-slave-jnhs krb5kdc[10287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434818, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:18.754053 10404 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:00:18.754446 10404 webserver.cc:460] Webserver started at http://127.8.102.1:45201/ using document root <none> and password file <none>
I20260926 15:00:18.754755 10404 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:18.754827 10404 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:18.754982 10404 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:18.756045 10404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "f35cb0916ab84cf8af024aaa83b2c74f"
format_stamp: "Formatted at 2026-09-26 15:00:18 on dist-test-slave-jnhs"
server_key: "3e17e2c7313ab8057f584717c7403e2a"
server_key_iv: "47ed9efe5ecbf2d437990837ba968637"
server_key_version: "encryptionkey@0"
I20260926 15:00:18.756481 10404 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "f35cb0916ab84cf8af024aaa83b2c74f"
format_stamp: "Formatted at 2026-09-26 15:00:18 on dist-test-slave-jnhs"
server_key: "3e17e2c7313ab8057f584717c7403e2a"
server_key_iv: "47ed9efe5ecbf2d437990837ba968637"
server_key_version: "encryptionkey@0"
I20260926 15:00:18.758189 10404 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 15:00:18.759121 10419 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:18.759366 10404 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 15:00:18.759541 10404 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "f35cb0916ab84cf8af024aaa83b2c74f"
format_stamp: "Formatted at 2026-09-26 15:00:18 on dist-test-slave-jnhs"
server_key: "3e17e2c7313ab8057f584717c7403e2a"
server_key_iv: "47ed9efe5ecbf2d437990837ba968637"
server_key_version: "encryptionkey@0"
I20260926 15:00:18.759639 10404 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:18.776741 10404 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:18.779568 10404 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:18.779752 10404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:18.780329 10404 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:18.780397 10404 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:18.780452 10404 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:18.780493 10404 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:18.787999 10404 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:42289
Sep 26 15:00:18 dist-test-slave-jnhs krb5kdc[10287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434818, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:18.788102 10534 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:42289 every 8 connection(s)
I20260926 15:00:18.788444 10404 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:00:18.789517 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 10404
I20260926 15:00:18.789652 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:00:18.789875 8600 external_mini_cluster.cc:1428] Setting key 143dc8ed1b10922f55726d3ded6a1400
I20260926 15:00:18.792215 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:18.781002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:50615 (local address 127.8.102.62:37383)
0926 15:00:18.781063 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:18.781066 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:18.781263 (+ 197us) server_negotiation.cc:407] Connection header received
0926 15:00:18.781865 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:18.781867 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:18.781879 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:18.781946 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:18.782863 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:18.783819 (+ 956us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:18.785413 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:18.785795 (+ 382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:18.785892 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:18.788669 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:18.788670 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:18.788671 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:18.788681 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:18.790481 (+ 1800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:18.790988 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:18.790990 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:18.790990 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:18.791062 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:18.791366 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:18.791368 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:18.791368 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:18.791591 (+ 223us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:18.791666 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:18.791963 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 15:00:18.792070 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 15:00:18.792389 10428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:18.780940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:37383 (local address 127.8.102.1:50615)
0926 15:00:18.781137 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:18.781175 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:18.781754 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:18.782031 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:18.782035 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:18.782415 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:18.782740 (+ 325us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:18.782749 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:18.783939 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:18.783940 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:18.785300 (+ 1360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:18.785303 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:18.785889 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:18.785890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:18.785998 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:18.786743 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:18.786752 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:18.788538 (+ 1786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:18.790616 (+ 2078us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:18.790624 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:18.790627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:18.790903 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:18.791154 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:18.791155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:18.791155 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:18.791277 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:18.791668 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:18.791675 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:18.791806 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 15:00:18.791919 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 15:00:18.792064 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":112,"threads_started":1}
I20260926 15:00:18.793087 10535 heartbeater.cc:347] Connected to a master server at 127.8.102.62:37383
I20260926 15:00:18.793222 10535 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:18.793474 10535 heartbeater.cc:511] Master 127.8.102.62:37383 requested a full tablet report, sending...
I20260926 15:00:18.794027 10331 ts_manager.cc:194] Registered new tserver with Master: f35cb0916ab84cf8af024aaa83b2c74f (127.8.102.1:42289)
I20260926 15:00:18.795079 10331 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:50615
WARNING: no policy specified for kudu/127.8.102.2@KRBTEST.COM; defaulting to no policy
I20260926 15:00:18.800688 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:18.794377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:57689 (local address 127.8.102.62:37383)
0926 15:00:18.794413 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:18.794415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:18.794423 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:00:18.794531 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:18.794533 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:18.794542 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:18.794604 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:18.795052 (+ 448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:18.795766 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:18.796667 (+ 901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:18.796874 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:18.796955 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:18.798079 (+ 1124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:18.798081 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:18.798081 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:18.798088 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:18.799569 (+ 1481us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:18.799926 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:18.799927 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:18.799928 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:18.799984 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:18.800168 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:18.800169 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:18.800170 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:18.800278 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:18.800353 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:18.800516 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 15:00:18.800589 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 15:00:18.800716 10542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:18.794200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:37383 (local address 127.8.102.1:57689)
0926 15:00:18.794366 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:18.794380 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:18.794450 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:18.794610 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:18.794611 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:18.794897 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:18.794972 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:18.794974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:18.795892 (+ 918us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:18.795894 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:18.796562 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:18.796564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:18.796953 (+ 389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:18.796954 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:18.797035 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:18.797638 (+ 603us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:18.797642 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:18.797930 (+ 288us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:18.799669 (+ 1739us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:18.799670 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:18.799671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:18.799841 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:18.800052 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:18.800053 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:18.800053 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:18.800118 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:18.800345 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:18.800346 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:18.800394 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 15:00:18.800484 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 15:00:18.800592 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":86,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
I20260926 15:00:18.818005 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37383
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:18.901996 10544 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:18.902295 10544 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:18.902343 10544 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:18.903688 10544 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:18.903771 10544 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:18.903810 10544 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 15:00:18.903844 10544 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:00:18.903887 10544 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:00:18.905514 10544 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.102.62:37383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10544
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:18.905826 10544 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:18.906095 10544 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:18.911760 10552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:18.911787 10550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:18.911767 10549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:18.912140 10544 server_base.cc:1034] running on GCE node
I20260926 15:00:18.912616 10544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:18.912963 10544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:18.914172 10544 hybrid_clock.cc:648] HybridClock initialized: now 1790434818914169 us; error 59 us; skew 500 ppm
Sep 26 15:00:18 dist-test-slave-jnhs krb5kdc[10287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434818, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:18.916908 10544 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:00:18.917364 10544 webserver.cc:460] Webserver started at http://127.8.102.2:36465/ using document root <none> and password file <none>
I20260926 15:00:18.917609 10544 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:18.917678 10544 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:18.917830 10544 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:18.918972 10544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "d7db7245daa640c588c327d148135a3f"
format_stamp: "Formatted at 2026-09-26 15:00:18 on dist-test-slave-jnhs"
server_key: "1fe2b088396b09787c3331e938b52765"
server_key_iv: "4cd801272ceca0b801b1961f36159373"
server_key_version: "encryptionkey@0"
I20260926 15:00:18.919328 10544 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "d7db7245daa640c588c327d148135a3f"
format_stamp: "Formatted at 2026-09-26 15:00:18 on dist-test-slave-jnhs"
server_key: "1fe2b088396b09787c3331e938b52765"
server_key_iv: "4cd801272ceca0b801b1961f36159373"
server_key_version: "encryptionkey@0"
I20260926 15:00:18.920696 10544 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:00:18.921487 10559 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:18.921657 10544 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 15:00:18.921759 10544 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "d7db7245daa640c588c327d148135a3f"
format_stamp: "Formatted at 2026-09-26 15:00:18 on dist-test-slave-jnhs"
server_key: "1fe2b088396b09787c3331e938b52765"
server_key_iv: "4cd801272ceca0b801b1961f36159373"
server_key_version: "encryptionkey@0"
I20260926 15:00:18.921842 10544 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:18.940570 10544 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:18.942785 10544 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:18.942972 10544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:18.943534 10544 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:18.943594 10544 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:18.943642 10544 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:18.943676 10544 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:00:18 dist-test-slave-jnhs krb5kdc[10287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434818, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:18.950524 10544 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:35893
I20260926 15:00:18.950608 10674 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:35893 every 8 connection(s)
I20260926 15:00:18.950969 10544 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:00:18.953266 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 10544
I20260926 15:00:18.953398 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:00:18.953620 8600 external_mini_cluster.cc:1428] Setting key 35c89aa21341235256191bc3129f0d4f
I20260926 15:00:18.954093 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:18.944228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:36445 (local address 127.8.102.62:37383)
0926 15:00:18.944284 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:18.944286 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:18.944495 (+ 209us) server_negotiation.cc:407] Connection header received
0926 15:00:18.945119 (+ 624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:18.945121 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:18.945133 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:18.945180 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:18.946094 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:18.946874 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:18.947953 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:18.948157 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:18.948244 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:18.951046 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:18.951048 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:18.951048 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:18.951056 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:18.952584 (+ 1528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:18.953026 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:18.953034 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:18.953035 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:18.953096 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:18.953354 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:18.953355 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:18.953355 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:18.953491 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:18.953583 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:18.953902 (+ 319us) server_negotiation.cc:299] Negotiation successful
0926 15:00:18.953987 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 15:00:18.954294 10568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:18.944198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:37383 (local address 127.8.102.2:36445)
0926 15:00:18.944373 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:18.944409 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:18.944998 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:18.945258 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:18.945262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:18.945660 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:18.945974 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:18.945981 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:18.947007 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:18.947008 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:18.947839 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:18.947841 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:18.948266 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:18.948267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:18.948438 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:18.949413 (+ 975us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:18.949421 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:18.950931 (+ 1510us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:18.952693 (+ 1762us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:18.952700 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:18.952702 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:18.952934 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:18.953167 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:18.953167 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:18.953168 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:18.953284 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:18.953573 (+ 289us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:18.953580 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:18.953757 (+ 177us) client_negotiation.cc:769] Sending connection context
0926 15:00:18.953882 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 15:00:18.954052 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260926 15:00:18.954782 10675 heartbeater.cc:347] Connected to a master server at 127.8.102.62:37383
I20260926 15:00:18.954864 10675 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:18.955093 10675 heartbeater.cc:511] Master 127.8.102.62:37383 requested a full tablet report, sending...
I20260926 15:00:18.955482 10332 ts_manager.cc:194] Registered new tserver with Master: d7db7245daa640c588c327d148135a3f (127.8.102.2:35893)
I20260926 15:00:18.955991 10332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:36445
I20260926 15:00:18.962157 10682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:18.955842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:37383 (local address 127.8.102.2:48477)
0926 15:00:18.956047 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:18.956059 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:18.956182 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:18.956361 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:18.956362 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:18.956538 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:18.956591 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:18.956592 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:18.957440 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:18.957440 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:18.957933 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:18.957935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:18.958424 (+ 489us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:18.958425 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:18.958479 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:18.958970 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:18.958976 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:18.959280 (+ 304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:18.961133 (+ 1853us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:18.961134 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:18.961134 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:18.961303 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:18.961511 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:18.961512 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:18.961512 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:18.961569 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:18.961903 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:18.961903 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:18.961941 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 15:00:18.961992 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 15:00:18.962086 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":88,"threads_started":1}
I20260926 15:00:18.962215 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:18.956018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:48477 (local address 127.8.102.62:37383)
0926 15:00:18.956055 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:18.956058 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:18.956109 (+ 51us) server_negotiation.cc:407] Connection header received
0926 15:00:18.956264 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:18.956265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:18.956277 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:18.956343 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:18.956685 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:18.957333 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:18.958014 (+ 681us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:18.958335 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:18.958409 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:18.959368 (+ 959us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:18.959370 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:18.959370 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:18.959380 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:18.961037 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:18.961379 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:18.961380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:18.961381 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:18.961442 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:18.961631 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:18.961632 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:18.961632 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:18.961834 (+ 202us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:18.961910 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:18.962026 (+ 116us) server_negotiation.cc:299] Negotiation successful
0926 15:00:18.962112 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
WARNING: no policy specified for kudu/127.8.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
I20260926 15:00:18.983150 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37383
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:38763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:19.065757 10684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:19.065959 10684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:19.065984 10684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:19.067140 10684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:19.067183 10684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:19.067204 10684 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260926 15:00:19.067215 10684 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:00:19.067233 10684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:00:19.068691 10684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:38763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.102.62:37383
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:19.068949 10684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:19.069186 10684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:19.074986 10690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:19.075001 10689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:19.075147 10684 server_base.cc:1034] running on GCE node
W20260926 15:00:19.075019 10692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:19.075459 10684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:19.075713 10684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:19.076884 10684 hybrid_clock.cc:648] HybridClock initialized: now 1790434819076889 us; error 49 us; skew 500 ppm
Sep 26 15:00:19 dist-test-slave-jnhs krb5kdc[10287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434819, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:19.079577 10684 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:00:19.080080 10684 webserver.cc:460] Webserver started at http://127.8.102.3:39323/ using document root <none> and password file <none>
I20260926 15:00:19.080317 10684 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:19.080384 10684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:19.080540 10684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:19.081600 10684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "138b3ab43d81457287449c23e0e4124e"
format_stamp: "Formatted at 2026-09-26 15:00:19 on dist-test-slave-jnhs"
server_key: "2a1eae35e42aea8456ea9e1c8604c585"
server_key_iv: "f9cb3fc39fa969c40fb5ec1dea8df44a"
server_key_version: "encryptionkey@0"
I20260926 15:00:19.081969 10684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "138b3ab43d81457287449c23e0e4124e"
format_stamp: "Formatted at 2026-09-26 15:00:19 on dist-test-slave-jnhs"
server_key: "2a1eae35e42aea8456ea9e1c8604c585"
server_key_iv: "f9cb3fc39fa969c40fb5ec1dea8df44a"
server_key_version: "encryptionkey@0"
I20260926 15:00:19.083410 10684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:00:19.084177 10699 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:19.084342 10684 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 15:00:19.084439 10684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "138b3ab43d81457287449c23e0e4124e"
format_stamp: "Formatted at 2026-09-26 15:00:19 on dist-test-slave-jnhs"
server_key: "2a1eae35e42aea8456ea9e1c8604c585"
server_key_iv: "f9cb3fc39fa969c40fb5ec1dea8df44a"
server_key_version: "encryptionkey@0"
I20260926 15:00:19.084569 10684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:19.110028 10684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:19.112422 10684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:19.112605 10684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:19.113224 10684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:19.113273 10684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:19.113304 10684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:19.113327 10684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:00:19 dist-test-slave-jnhs krb5kdc[10287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434819, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:19.120700 10684 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:42727
I20260926 15:00:19.120839 10814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:42727 every 8 connection(s)
I20260926 15:00:19.121111 10684 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:00:19.123644 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.113659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:42495 (local address 127.8.102.62:37383)
0926 15:00:19.113709 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:19.113711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:19.113937 (+ 226us) server_negotiation.cc:407] Connection header received
0926 15:00:19.114473 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:19.114475 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:19.114488 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:19.114538 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:19.115288 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.115798 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.116881 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.117077 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.117155 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.120338 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:19.120339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:19.120340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:19.120348 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:19.122150 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:19.122673 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:19.122674 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:19.122674 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:19.122739 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:19.122991 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:19.122993 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:19.122993 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:19.123116 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:19.123190 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:19.123462 (+ 272us) server_negotiation.cc:299] Negotiation successful
0926 15:00:19.123552 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 15:00:19.123840 10708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.113640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:37383 (local address 127.8.102.3:42495)
0926 15:00:19.113809 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:19.113849 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:19.114367 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:19.114612 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:19.114616 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:19.114981 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:19.115185 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.115191 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.115897 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.115897 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.116776 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.116778 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.117163 (+ 385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.117164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.117290 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.117864 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:19.117873 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:19.120247 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:19.122246 (+ 1999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:19.122252 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:19.122255 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:19.122561 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:19.122805 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:19.122806 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:19.122807 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:19.122920 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:19.123182 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:19.123188 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:19.123333 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 15:00:19.123445 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 15:00:19.123607 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":81,"threads_started":1}
I20260926 15:00:19.124357 10815 heartbeater.cc:347] Connected to a master server at 127.8.102.62:37383
I20260926 15:00:19.124439 10815 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:19.124620 10815 heartbeater.cc:511] Master 127.8.102.62:37383 requested a full tablet report, sending...
I20260926 15:00:19.125026 10332 ts_manager.cc:194] Registered new tserver with Master: 138b3ab43d81457287449c23e0e4124e (127.8.102.3:42727)
I20260926 15:00:19.125455 10332 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:42495
I20260926 15:00:19.128607 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 10684
I20260926 15:00:19.128710 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:00:19.128895 8600 external_mini_cluster.cc:1428] Setting key 0034841fce00c0ae7cc0b436ac2eefaf
I20260926 15:00:19.130107 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:00:19.131574 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.125416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:52991 (local address 127.8.102.62:37383)
0926 15:00:19.125483 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:19.125485 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:19.125590 (+ 105us) server_negotiation.cc:407] Connection header received
0926 15:00:19.125672 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:19.125672 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:19.125682 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:19.125738 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:19.126104 (+ 366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.126610 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.127320 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.127491 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.127577 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.128687 (+ 1110us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:19.128689 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:19.128689 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:19.128696 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:19.130368 (+ 1672us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:19.130842 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:19.130843 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:19.130843 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:19.130894 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:19.131108 (+ 214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:19.131109 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:19.131109 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:19.131200 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:19.131245 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:19.131435 (+ 190us) server_negotiation.cc:299] Negotiation successful
0926 15:00:19.131489 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 15:00:19.132017 10821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.125363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:37383 (local address 127.8.102.3:52991)
0926 15:00:19.125519 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:19.125530 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:19.125616 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:19.125741 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:19.125742 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:19.125955 (+ 213us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:19.126026 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.126027 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.126733 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.126734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.127241 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.127243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.127582 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.127583 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.127665 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.128217 (+ 552us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:19.128222 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:19.128560 (+ 338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:19.130486 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:19.130487 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:19.130487 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:19.130765 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:19.130963 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:19.130964 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:19.130965 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:19.131045 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:19.131261 (+ 216us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:19.131262 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:19.131320 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 15:00:19.131420 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 15:00:19.131552 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
Sep 26 15:00:19 dist-test-slave-jnhs krb5kdc[10287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 15:00:19 dist-test-slave-jnhs krb5kdc[10287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434819, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:19.155447 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.147832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51502 (local address 127.8.102.62:37383)
0926 15:00:19.147945 (+ 113us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:19.147948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:19.147960 (+ 12us) server_negotiation.cc:407] Connection header received
0926 15:00:19.148029 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:19.148030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:19.148040 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:19.148105 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:19.148771 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.149285 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.150067 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.150264 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.150325 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.152792 (+ 2467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:19.152793 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:19.152793 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:19.152800 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:19.154299 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:19.154585 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:19.154585 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:19.154586 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:19.154676 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:19.154924 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:19.154925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:19.154925 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:19.155037 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:19.155146 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:19.155255 (+ 109us) server_negotiation.cc:299] Negotiation successful
0926 15:00:19.155339 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260926 15:00:19.157410 10332 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:00:19.159325 10332 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:00:19.165961 10832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.161843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:42289 (local address 127.0.0.1:59434)
0926 15:00:19.162479 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:19.162490 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:19.162544 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:19.162812 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:19.162813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:19.162815 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:19.162858 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.162859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.163943 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.163944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.164866 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.164868 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.165643 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.165644 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.165721 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.165736 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 15:00:19.165814 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 15:00:19.165866 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"mutex_wait_us":436,"thread_start_us":76,"threads_started":1}
I20260926 15:00:19.165990 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.161993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59434 (local address 127.8.102.1:42289)
0926 15:00:19.162433 (+ 440us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:19.162437 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:19.162600 (+ 163us) server_negotiation.cc:407] Connection header received
0926 15:00:19.162662 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:19.162664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:19.162680 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:19.162749 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:19.162953 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.163863 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.164962 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.165559 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.165639 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.165700 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:19.165839 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 15:00:19.165879 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":406,"thread_start_us":77,"threads_started":1}
I20260926 15:00:19.166544 10837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.161843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:35893 (local address 127.0.0.1:49260)
0926 15:00:19.162416 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:19.162426 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:19.162584 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:19.162952 (+ 368us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:19.162953 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:19.162954 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:19.162997 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.162998 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.164225 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.164226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.165301 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.165304 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.166310 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.166311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.166371 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.166375 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:00:19.166439 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 15:00:19.166475 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"mutex_wait_us":5,"thread_start_us":123,"threads_started":1}
I20260926 15:00:19.166625 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.162129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49260 (local address 127.8.102.2:35893)
0926 15:00:19.162303 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:19.162307 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:19.162552 (+ 245us) server_negotiation.cc:407] Connection header received
0926 15:00:19.162718 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:19.162721 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:19.162738 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:19.162793 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:19.163081 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.164145 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.165403 (+ 1258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.166223 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.166302 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.166358 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:19.166459 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 15:00:19.166533 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":79,"threads_started":1}
I20260926 15:00:19.166952 10469 tablet_service.cc:1467] Processing CreateTablet for tablet a9d30bf600fa4ba786b9a93e8ee9c4e2 (DEFAULT_TABLE table=test-table [id=03307f867bb64f62b59fe0b152741d82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:19.167158 10833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.161842 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:42727 (local address 127.0.0.1:46754)
0926 15:00:19.162039 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:19.162083 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:19.162159 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:19.162397 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:19.162400 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:19.162411 (+ 11us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:19.162493 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.162496 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.163886 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.163887 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.165458 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.165460 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.166877 (+ 1417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.166878 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.167011 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.167015 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:00:19.167066 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 15:00:19.167103 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":162,"mutex_wait_us":37,"thread_start_us":57,"threads_started":1}
I20260926 15:00:19.167371 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d30bf600fa4ba786b9a93e8ee9c4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:19.167477 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.161955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46754 (local address 127.8.102.3:42727)
0926 15:00:19.162193 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:19.162198 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:19.162210 (+ 12us) server_negotiation.cc:407] Connection header received
0926 15:00:19.162251 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:19.162253 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:19.162272 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:19.162495 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:19.162561 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.163804 (+ 1243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.165544 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.166570 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.166876 (+ 306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.166937 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:19.167332 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 15:00:19.167371 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":199,"thread_start_us":86,"threads_started":1}
I20260926 15:00:19.168001 10749 tablet_service.cc:1467] Processing CreateTablet for tablet a9d30bf600fa4ba786b9a93e8ee9c4e2 (DEFAULT_TABLE table=test-table [id=03307f867bb64f62b59fe0b152741d82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:19.168275 10749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d30bf600fa4ba786b9a93e8ee9c4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:19.170598 10838 tablet_bootstrap.cc:492] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f: Bootstrap starting.
I20260926 15:00:19.170955 10839 tablet_bootstrap.cc:492] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e: Bootstrap starting.
I20260926 15:00:19.171409 10838 tablet_bootstrap.cc:654] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:19.171300 10609 tablet_service.cc:1467] Processing CreateTablet for tablet a9d30bf600fa4ba786b9a93e8ee9c4e2 (DEFAULT_TABLE table=test-table [id=03307f867bb64f62b59fe0b152741d82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:19.171650 10609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a9d30bf600fa4ba786b9a93e8ee9c4e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:19.171739 10839 tablet_bootstrap.cc:654] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:19.171769 10838 log.cc:824] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:19.172101 10839 log.cc:824] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:19.172585 10838 tablet_bootstrap.cc:492] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f: No bootstrap required, opened a new log
I20260926 15:00:19.172677 10838 ts_tablet_manager.cc:1397] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 15:00:19.174422 10838 raft_consensus.cc:348] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } }
I20260926 15:00:19.174585 10838 raft_consensus.cc:374] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:19.174628 10838 raft_consensus.cc:729] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f35cb0916ab84cf8af024aaa83b2c74f, State: Initialized, Role: FOLLOWER
I20260926 15:00:19.174795 10842 tablet_bootstrap.cc:492] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f: Bootstrap starting.
I20260926 15:00:19.174839 10838 consensus_queue.cc:260] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } }
I20260926 15:00:19.175096 10838 ts_tablet_manager.cc:1428] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260926 15:00:19.175253 10535 heartbeater.cc:503] Master 127.8.102.62:37383 was elected leader, sending a full tablet report...
I20260926 15:00:19.175292 10844 raft_consensus.cc:482] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:19.175351 10844 raft_consensus.cc:504] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } }
I20260926 15:00:19.175402 10839 tablet_bootstrap.cc:492] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e: No bootstrap required, opened a new log
I20260926 15:00:19.175467 10842 tablet_bootstrap.cc:654] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:19.175484 10839 ts_tablet_manager.cc:1397] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 15:00:19.175804 10844 leader_election.cc:288] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 138b3ab43d81457287449c23e0e4124e (127.8.102.3:42727), d7db7245daa640c588c327d148135a3f (127.8.102.2:35893)
I20260926 15:00:19.175805 10842 log.cc:824] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:19.177630 10839 raft_consensus.cc:348] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } }
I20260926 15:00:19.177800 10839 raft_consensus.cc:374] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:19.177837 10839 raft_consensus.cc:729] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138b3ab43d81457287449c23e0e4124e, State: Initialized, Role: FOLLOWER
I20260926 15:00:19.177953 10839 consensus_queue.cc:260] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } }
I20260926 15:00:19.178213 10839 ts_tablet_manager.cc:1428] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:00:19.178236 10815 heartbeater.cc:503] Master 127.8.102.62:37383 was elected leader, sending a full tablet report...
I20260926 15:00:19.178874 10842 tablet_bootstrap.cc:492] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f: No bootstrap required, opened a new log
I20260926 15:00:19.178953 10842 ts_tablet_manager.cc:1397] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 15:00:19.179479 10428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.175791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:42727 (local address 127.8.102.1:36465)
0926 15:00:19.175985 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:19.175998 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:19.176038 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:19.176241 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:19.176242 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:19.176244 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:19.176308 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.176310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.177231 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.177232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.178406 (+ 1174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.178409 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.179203 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.179205 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.179282 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.179285 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:00:19.179344 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 15:00:19.179399 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":174}
I20260926 15:00:19.179486 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.175986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:36465 (local address 127.8.102.3:42727)
0926 15:00:19.176126 (+ 140us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:19.176128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:19.176134 (+ 6us) server_negotiation.cc:407] Connection header received
0926 15:00:19.176154 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:19.176155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:19.176163 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:19.176203 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:19.176428 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.177128 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.178514 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.179088 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.179158 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.179170 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:19.179356 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 15:00:19.179396 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260926 15:00:19.180343 10769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d30bf600fa4ba786b9a93e8ee9c4e2" candidate_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138b3ab43d81457287449c23e0e4124e" is_pre_election: true
I20260926 15:00:19.180490 10769 raft_consensus.cc:2454] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f35cb0916ab84cf8af024aaa83b2c74f in term 0.
I20260926 15:00:19.180500 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.176498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:44127 (local address 127.8.102.2:35893)
0926 15:00:19.176538 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:19.176540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:19.176698 (+ 158us) server_negotiation.cc:407] Connection header received
0926 15:00:19.176871 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:19.176872 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:19.176881 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:19.176921 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:19.177137 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.177716 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.179507 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.180009 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.180046 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.180054 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:19.180386 (+ 332us) server_negotiation.cc:299] Negotiation successful
0926 15:00:19.180436 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 15:00:19.180521 10846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.175791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:35893 (local address 127.8.102.1:44127)
0926 15:00:19.176635 (+ 844us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:19.176646 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:19.176792 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:19.176988 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:19.176989 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:19.176990 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:19.177056 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.177057 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.177808 (+ 751us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.177809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.179414 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.179416 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.180239 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.180240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.180313 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.180316 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:00:19.180381 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 15:00:19.180420 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":828,"mutex_wait_us":527,"thread_start_us":91,"threads_started":1}
I20260926 15:00:19.180713 10421 leader_election.cc:302] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138b3ab43d81457287449c23e0e4124e, f35cb0916ab84cf8af024aaa83b2c74f; no voters:
I20260926 15:00:19.180845 10843 raft_consensus.cc:2781] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:00:19.180760 10842 raft_consensus.cc:348] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } }
I20260926 15:00:19.180926 10843 raft_consensus.cc:482] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:00:19.180932 10842 raft_consensus.cc:374] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:19.180915 10629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d30bf600fa4ba786b9a93e8ee9c4e2" candidate_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7db7245daa640c588c327d148135a3f" is_pre_election: true
I20260926 15:00:19.180960 10843 raft_consensus.cc:3037] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:19.180964 10842 raft_consensus.cc:729] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7db7245daa640c588c327d148135a3f, State: Initialized, Role: FOLLOWER
I20260926 15:00:19.181073 10842 consensus_queue.cc:260] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } }
I20260926 15:00:19.181167 10629 raft_consensus.cc:2454] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f35cb0916ab84cf8af024aaa83b2c74f in term 0.
I20260926 15:00:19.181396 10842 ts_tablet_manager.cc:1428] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260926 15:00:19.181401 10675 heartbeater.cc:503] Master 127.8.102.62:37383 was elected leader, sending a full tablet report...
I20260926 15:00:19.182281 10843 raft_consensus.cc:504] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } }
I20260926 15:00:19.182500 10843 leader_election.cc:288] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [CANDIDATE]: Term 1 election: Requested vote from peers 138b3ab43d81457287449c23e0e4124e (127.8.102.3:42727), d7db7245daa640c588c327d148135a3f (127.8.102.2:35893)
I20260926 15:00:19.182602 10769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d30bf600fa4ba786b9a93e8ee9c4e2" candidate_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138b3ab43d81457287449c23e0e4124e"
I20260926 15:00:19.182603 10629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a9d30bf600fa4ba786b9a93e8ee9c4e2" candidate_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7db7245daa640c588c327d148135a3f"
I20260926 15:00:19.182706 10629 raft_consensus.cc:3037] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:19.182708 10769 raft_consensus.cc:3037] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:19.183564 10769 raft_consensus.cc:2454] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f35cb0916ab84cf8af024aaa83b2c74f in term 1.
I20260926 15:00:19.183564 10629 raft_consensus.cc:2454] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f35cb0916ab84cf8af024aaa83b2c74f in term 1.
I20260926 15:00:19.183728 10421 leader_election.cc:302] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138b3ab43d81457287449c23e0e4124e, f35cb0916ab84cf8af024aaa83b2c74f; no voters:
I20260926 15:00:19.183856 10843 raft_consensus.cc:2781] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:19.183928 10843 raft_consensus.cc:686] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [term 1 LEADER]: Becoming Leader. State: Replica: f35cb0916ab84cf8af024aaa83b2c74f, State: Running, Role: LEADER
I20260926 15:00:19.184003 10843 consensus_queue.cc:237] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } }
I20260926 15:00:19.184768 10332 catalog_manager.cc:5495] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f reported cstate change: term changed from 0 to 1, leader changed from <none> to f35cb0916ab84cf8af024aaa83b2c74f (127.8.102.1). New cstate: current_term: 1 leader_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } health_report { overall_health: UNKNOWN } } }
W20260926 15:00:19.201642 10676 tablet.cc:2369] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 26 15:00:19 dist-test-slave-jnhs krb5kdc[10287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434818, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:19.219118 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.211486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51508 (local address 127.8.102.62:37383)
0926 15:00:19.211562 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:19.211565 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:19.211717 (+ 152us) server_negotiation.cc:407] Connection header received
0926 15:00:19.211769 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:19.211770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:19.211779 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:19.211842 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:19.212316 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.212799 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.213590 (+ 791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.213766 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.213823 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.216206 (+ 2383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:19.216207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:19.216207 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:19.216216 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:19.217727 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:19.218167 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:19.218169 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:19.218169 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:19.218231 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:19.218428 (+ 197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:19.218429 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:19.218429 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:19.218547 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:19.218665 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:19.218840 (+ 175us) server_negotiation.cc:299] Negotiation successful
0926 15:00:19.218972 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":51}
I20260926 15:00:19.219180 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.211396 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:37383 (local address 127.0.0.1:51508)
0926 15:00:19.211640 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:19.211651 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:19.211723 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:19.211857 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:19.211857 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:19.212175 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:19.212225 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.212227 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.212928 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.212929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.213505 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.213507 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.213850 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.213851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.213898 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.214533 (+ 635us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:19.214537 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:19.216093 (+ 1556us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:19.217849 (+ 1756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:19.217858 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:19.217861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:19.218085 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:19.218303 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:19.218304 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:19.218304 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:19.218365 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:19.218658 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:19.218666 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:19.218715 (+ 49us) client_negotiation.cc:769] Sending connection context
0926 15:00:19.218811 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 15:00:19.219032 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":209,"mutex_wait_us":106,"thread_start_us":164,"threads_started":1}
I20260926 15:00:19.221460 10332 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:51508:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 15:00:19.221761 10332 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:00:19.223682 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 913bd0135bee4e55a222e9e05209f7b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=baecbe9e30794ead851a2503bc51f9d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:00:19.223699 10609 tablet_service.cc:1467] Processing CreateTablet for tablet 913bd0135bee4e55a222e9e05209f7b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=baecbe9e30794ead851a2503bc51f9d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:00:19.223680 10749 tablet_service.cc:1467] Processing CreateTablet for tablet 913bd0135bee4e55a222e9e05209f7b0 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=baecbe9e30794ead851a2503bc51f9d8]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:00:19.223886 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913bd0135bee4e55a222e9e05209f7b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:19.223884 10609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913bd0135bee4e55a222e9e05209f7b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:19.223886 10749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 913bd0135bee4e55a222e9e05209f7b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:19.225528 10838 tablet_bootstrap.cc:492] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f: Bootstrap starting.
I20260926 15:00:19.225953 10838 tablet_bootstrap.cc:654] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:19.226336 10839 tablet_bootstrap.cc:492] T 913bd0135bee4e55a222e9e05209f7b0 P 138b3ab43d81457287449c23e0e4124e: Bootstrap starting.
I20260926 15:00:19.226872 10838 tablet_bootstrap.cc:492] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f: No bootstrap required, opened a new log
I20260926 15:00:19.226927 10838 ts_tablet_manager.cc:1397] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.001s
I20260926 15:00:19.226933 10842 tablet_bootstrap.cc:492] T 913bd0135bee4e55a222e9e05209f7b0 P d7db7245daa640c588c327d148135a3f: Bootstrap starting.
I20260926 15:00:19.226967 10839 tablet_bootstrap.cc:654] T 913bd0135bee4e55a222e9e05209f7b0 P 138b3ab43d81457287449c23e0e4124e: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:19.227088 10838 raft_consensus.cc:348] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:19.227180 10838 raft_consensus.cc:374] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:19.227205 10838 raft_consensus.cc:729] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f35cb0916ab84cf8af024aaa83b2c74f, State: Initialized, Role: FOLLOWER
I20260926 15:00:19.227254 10838 consensus_queue.cc:260] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:19.227370 10838 ts_tablet_manager.cc:1428] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:19.227581 10842 tablet_bootstrap.cc:654] T 913bd0135bee4e55a222e9e05209f7b0 P d7db7245daa640c588c327d148135a3f: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:19.227906 10839 tablet_bootstrap.cc:492] T 913bd0135bee4e55a222e9e05209f7b0 P 138b3ab43d81457287449c23e0e4124e: No bootstrap required, opened a new log
I20260926 15:00:19.227984 10839 ts_tablet_manager.cc:1397] T 913bd0135bee4e55a222e9e05209f7b0 P 138b3ab43d81457287449c23e0e4124e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:00:19.228046 10843 tablet_replica.cc:439] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } } }
I20260926 15:00:19.228154 10843 tablet_replica.cc:442] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:00:19.228150 10839 raft_consensus.cc:348] T 913bd0135bee4e55a222e9e05209f7b0 P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:19.228217 10839 raft_consensus.cc:374] T 913bd0135bee4e55a222e9e05209f7b0 P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:19.228240 10839 raft_consensus.cc:729] T 913bd0135bee4e55a222e9e05209f7b0 P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138b3ab43d81457287449c23e0e4124e, State: Initialized, Role: FOLLOWER
I20260926 15:00:19.228276 10839 consensus_queue.cc:260] T 913bd0135bee4e55a222e9e05209f7b0 P 138b3ab43d81457287449c23e0e4124e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:19.228367 10839 ts_tablet_manager.cc:1428] T 913bd0135bee4e55a222e9e05209f7b0 P 138b3ab43d81457287449c23e0e4124e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:19.228497 10847 tablet_replica.cc:439] T 913bd0135bee4e55a222e9e05209f7b0 P 138b3ab43d81457287449c23e0e4124e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } } }
I20260926 15:00:19.228601 10847 tablet_replica.cc:442] T 913bd0135bee4e55a222e9e05209f7b0 P 138b3ab43d81457287449c23e0e4124e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:00:19.229061 10842 tablet_bootstrap.cc:492] T 913bd0135bee4e55a222e9e05209f7b0 P d7db7245daa640c588c327d148135a3f: No bootstrap required, opened a new log
I20260926 15:00:19.229146 10842 ts_tablet_manager.cc:1397] T 913bd0135bee4e55a222e9e05209f7b0 P d7db7245daa640c588c327d148135a3f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 15:00:19.229305 10842 raft_consensus.cc:348] T 913bd0135bee4e55a222e9e05209f7b0 P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:19.229365 10842 raft_consensus.cc:374] T 913bd0135bee4e55a222e9e05209f7b0 P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:19.229380 10842 raft_consensus.cc:729] T 913bd0135bee4e55a222e9e05209f7b0 P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7db7245daa640c588c327d148135a3f, State: Initialized, Role: FOLLOWER
I20260926 15:00:19.229439 10842 consensus_queue.cc:260] T 913bd0135bee4e55a222e9e05209f7b0 P d7db7245daa640c588c327d148135a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:19.229542 10842 ts_tablet_manager.cc:1428] T 913bd0135bee4e55a222e9e05209f7b0 P d7db7245daa640c588c327d148135a3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:19.229610 10848 tablet_replica.cc:439] T 913bd0135bee4e55a222e9e05209f7b0 P d7db7245daa640c588c327d148135a3f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } } }
I20260926 15:00:19.229727 10848 tablet_replica.cc:442] T 913bd0135bee4e55a222e9e05209f7b0 P d7db7245daa640c588c327d148135a3f: This TxnStatusTablet replica's current role is: FOLLOWER
W20260926 15:00:19.289399 10536 tablet.cc:2369] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:00:19.370277 10843 raft_consensus.cc:482] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:19.370507 10843 raft_consensus.cc:504] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:19.370775 10843 leader_election.cc:288] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7db7245daa640c588c327d148135a3f (127.8.102.2:35893), 138b3ab43d81457287449c23e0e4124e (127.8.102.3:42727)
I20260926 15:00:19.370991 10629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913bd0135bee4e55a222e9e05209f7b0" candidate_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7db7245daa640c588c327d148135a3f" is_pre_election: true
I20260926 15:00:19.370991 10769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913bd0135bee4e55a222e9e05209f7b0" candidate_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138b3ab43d81457287449c23e0e4124e" is_pre_election: true
I20260926 15:00:19.371088 10769 raft_consensus.cc:2454] T 913bd0135bee4e55a222e9e05209f7b0 P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f35cb0916ab84cf8af024aaa83b2c74f in term 0.
I20260926 15:00:19.371088 10629 raft_consensus.cc:2454] T 913bd0135bee4e55a222e9e05209f7b0 P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f35cb0916ab84cf8af024aaa83b2c74f in term 0.
I20260926 15:00:19.371222 10421 leader_election.cc:302] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138b3ab43d81457287449c23e0e4124e, f35cb0916ab84cf8af024aaa83b2c74f; no voters:
I20260926 15:00:19.371356 10843 raft_consensus.cc:2781] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:00:19.371402 10843 raft_consensus.cc:482] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:00:19.371417 10843 raft_consensus.cc:3037] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Advancing to term 1
W20260926 15:00:19.371693 10816 tablet.cc:2369] T 913bd0135bee4e55a222e9e05209f7b0 P 138b3ab43d81457287449c23e0e4124e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:00:19.372205 10843 raft_consensus.cc:504] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:19.372340 10843 leader_election.cc:288] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f [CANDIDATE]: Term 1 election: Requested vote from peers d7db7245daa640c588c327d148135a3f (127.8.102.2:35893), 138b3ab43d81457287449c23e0e4124e (127.8.102.3:42727)
I20260926 15:00:19.372556 10769 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913bd0135bee4e55a222e9e05209f7b0" candidate_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138b3ab43d81457287449c23e0e4124e"
I20260926 15:00:19.372561 10629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "913bd0135bee4e55a222e9e05209f7b0" candidate_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7db7245daa640c588c327d148135a3f"
I20260926 15:00:19.372643 10629 raft_consensus.cc:3037] T 913bd0135bee4e55a222e9e05209f7b0 P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:19.372642 10769 raft_consensus.cc:3037] T 913bd0135bee4e55a222e9e05209f7b0 P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:19.373474 10769 raft_consensus.cc:2454] T 913bd0135bee4e55a222e9e05209f7b0 P 138b3ab43d81457287449c23e0e4124e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f35cb0916ab84cf8af024aaa83b2c74f in term 1.
I20260926 15:00:19.373490 10629 raft_consensus.cc:2454] T 913bd0135bee4e55a222e9e05209f7b0 P d7db7245daa640c588c327d148135a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f35cb0916ab84cf8af024aaa83b2c74f in term 1.
I20260926 15:00:19.373663 10423 leader_election.cc:302] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7db7245daa640c588c327d148135a3f, f35cb0916ab84cf8af024aaa83b2c74f; no voters:
I20260926 15:00:19.373755 10843 raft_consensus.cc:2781] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:19.373857 10843 raft_consensus.cc:686] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f [term 1 LEADER]: Becoming Leader. State: Replica: f35cb0916ab84cf8af024aaa83b2c74f, State: Running, Role: LEADER
I20260926 15:00:19.373944 10843 consensus_queue.cc:237] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:19.374260 10844 tablet_replica.cc:439] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f: TxnStatusTablet state changed. Reason: New leader f35cb0916ab84cf8af024aaa83b2c74f. Latest consensus state: current_term: 1 leader_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } } }
I20260926 15:00:19.374372 10844 tablet_replica.cc:442] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:00:19.374629 10860 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 15:00:19.374687 10332 catalog_manager.cc:5495] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f reported cstate change: term changed from 0 to 1, leader changed from <none> to f35cb0916ab84cf8af024aaa83b2c74f (127.8.102.1). New cstate: current_term: 1 leader_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } health_report { overall_health: UNKNOWN } } }
I20260926 15:00:19.374837 10860 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 15:00:19.411947 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.409342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59446 (local address 127.8.102.1:42289)
0926 15:00:19.409413 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:19.409415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:19.409425 (+ 10us) server_negotiation.cc:407] Connection header received
0926 15:00:19.409488 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:19.409489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:19.409517 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:19.409592 (+ 75us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:19.409743 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.410256 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.411099 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.411298 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.411373 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.411582 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:19.411688 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:19.411783 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:19.411826 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 15:00:19.411849 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 15:00:19.411947 10855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.409244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:42289 (local address 127.0.0.1:59446)
0926 15:00:19.409319 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:19.409331 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:19.409399 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:19.409597 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:19.409597 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:19.409599 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:00:19.409656 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.409658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.410351 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.410353 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.411011 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.411014 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.411386 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.411387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.411453 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.411469 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:00:19.411758 (+ 289us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:00:19.411769 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 15:00:19.411819 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 15:00:19.411870 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 15:00:19.413949 10629 raft_consensus.cc:1261] T 913bd0135bee4e55a222e9e05209f7b0 P d7db7245daa640c588c327d148135a3f [term 1 FOLLOWER]: Refusing update from remote peer f35cb0916ab84cf8af024aaa83b2c74f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:19.413980 10769 raft_consensus.cc:1261] T 913bd0135bee4e55a222e9e05209f7b0 P 138b3ab43d81457287449c23e0e4124e [term 1 FOLLOWER]: Refusing update from remote peer f35cb0916ab84cf8af024aaa83b2c74f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:19.414348 10844 consensus_queue.cc:1035] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f [LEADER]: Connected to new peer: Peer: permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:19.414505 10843 consensus_queue.cc:1035] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:19.415726 10848 tablet_replica.cc:439] T 913bd0135bee4e55a222e9e05209f7b0 P d7db7245daa640c588c327d148135a3f: TxnStatusTablet state changed. Reason: New leader f35cb0916ab84cf8af024aaa83b2c74f. Latest consensus state: current_term: 1 leader_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } } }
I20260926 15:00:19.415843 10848 tablet_replica.cc:442] T 913bd0135bee4e55a222e9e05209f7b0 P d7db7245daa640c588c327d148135a3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:00:19.415994 10847 tablet_replica.cc:439] T 913bd0135bee4e55a222e9e05209f7b0 P 138b3ab43d81457287449c23e0e4124e: TxnStatusTablet state changed. Reason: New leader f35cb0916ab84cf8af024aaa83b2c74f. Latest consensus state: current_term: 1 leader_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } } }
I20260926 15:00:19.416117 10847 tablet_replica.cc:442] T 913bd0135bee4e55a222e9e05209f7b0 P 138b3ab43d81457287449c23e0e4124e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:00:19.416644 10863 mvcc.cc:205] Tried to move back new op lower bound from 7333621020316499968 to 7333621020156157952. Current Snapshot: MvccSnapshot[applied={T|T < 7333621020316499968}]
I20260926 15:00:19.416716 10843 tablet_replica.cc:439] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } } }
I20260926 15:00:19.416780 10843 tablet_replica.cc:442] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:00:19.417266 10865 mvcc.cc:205] Tried to move back new op lower bound from 7333621020316499968 to 7333621020156157952. Current Snapshot: MvccSnapshot[applied={T|T < 7333621020316499968}]
I20260926 15:00:19.417395 10847 tablet_replica.cc:439] T 913bd0135bee4e55a222e9e05209f7b0 P 138b3ab43d81457287449c23e0e4124e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } } }
I20260926 15:00:19.417465 10847 tablet_replica.cc:442] T 913bd0135bee4e55a222e9e05209f7b0 P 138b3ab43d81457287449c23e0e4124e: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:00:19.417945 10844 tablet_replica.cc:439] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } } }
I20260926 15:00:19.418041 10844 tablet_replica.cc:442] T 913bd0135bee4e55a222e9e05209f7b0 P f35cb0916ab84cf8af024aaa83b2c74f: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:00:19.418543 10848 tablet_replica.cc:439] T 913bd0135bee4e55a222e9e05209f7b0 P d7db7245daa640c588c327d148135a3f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } } }
I20260926 15:00:19.418658 10848 tablet_replica.cc:442] T 913bd0135bee4e55a222e9e05209f7b0 P d7db7245daa640c588c327d148135a3f: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:00:19.418581 10864 mvcc.cc:205] Tried to move back new op lower bound from 7333621020316499968 to 7333621020156157952. Current Snapshot: MvccSnapshot[applied={T|T < 7333621020316499968}]
I20260926 15:00:19.423395 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.420705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59462 (local address 127.8.102.1:42289)
0926 15:00:19.420763 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:19.420765 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:19.420774 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:19.420805 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:19.420805 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:19.420814 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:19.420860 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:19.421037 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.421590 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.422727 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.422919 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.422978 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.423144 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:19.423170 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:19.423204 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:19.423283 (+ 79us) server_negotiation.cc:299] Negotiation successful
0926 15:00:19.423301 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 15:00:19.429255 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.426426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:40941 (local address 127.8.102.1:42289)
0926 15:00:19.426470 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:19.426472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:19.426590 (+ 118us) server_negotiation.cc:407] Connection header received
0926 15:00:19.426620 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:19.426620 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:19.426628 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:19.426713 (+ 85us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:19.426872 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.427347 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.428454 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.428662 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.428718 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.428943 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:19.428975 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:19.429031 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:19.429112 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 15:00:19.429149 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":24}
I20260926 15:00:19.429335 10874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.426337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:42289 (local address 127.8.102.1:40941)
0926 15:00:19.426491 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:19.426503 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:19.426568 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:19.426717 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:19.426718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:19.426719 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:00:19.426780 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.426781 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.427423 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.427424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.428365 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.428367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.428738 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.428739 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.428805 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.428821 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:00:19.429054 (+ 233us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:00:19.429055 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 15:00:19.429109 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 15:00:19.429172 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":4,"thread_start_us":93,"threads_started":1}
I20260926 15:00:19.431532 10629 raft_consensus.cc:1261] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f [term 1 FOLLOWER]: Refusing update from remote peer f35cb0916ab84cf8af024aaa83b2c74f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:19.431543 10769 raft_consensus.cc:1261] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e [term 1 FOLLOWER]: Refusing update from remote peer f35cb0916ab84cf8af024aaa83b2c74f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:19.431767 10843 consensus_queue.cc:1035] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:19.431859 10844 consensus_queue.cc:1035] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [LEADER]: Connected to new peer: Peer: permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:19.443924 10332 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:19.444065 10332 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:19.444923 10332 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78: Sending DeleteTablet for 3 replicas of tablet a9d30bf600fa4ba786b9a93e8ee9c4e2
I20260926 15:00:19.445221 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet a9d30bf600fa4ba786b9a93e8ee9c4e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:19 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:59434
I20260926 15:00:19.445292 10749 tablet_service.cc:1514] Processing DeleteTablet for tablet a9d30bf600fa4ba786b9a93e8ee9c4e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:19 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:46754
I20260926 15:00:19.445617 10880 tablet_replica.cc:330] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e: stopping tablet replica
I20260926 15:00:19.445623 10609 tablet_service.cc:1514] Processing DeleteTablet for tablet a9d30bf600fa4ba786b9a93e8ee9c4e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:19 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:49260
I20260926 15:00:19.445623 10332 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 15:00:19.445737 10880 raft_consensus.cc:2229] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e [term 1 FOLLOWER]: Raft consensus shutting down.
W20260926 15:00:19.445766 10332 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:00:19.445796 10880 raft_consensus.cc:2258] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:19.445839 10879 tablet_replica.cc:330] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f: stopping tablet replica
I20260926 15:00:19.445935 10879 raft_consensus.cc:2229] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:00:19.446026 10879 raft_consensus.cc:2258] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:19.446259 10880 ts_tablet_manager.cc:1905] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:19.447108 10881 tablet_replica.cc:330] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f: stopping tablet replica
I20260926 15:00:19.447170 10749 tablet_service.cc:1467] Processing CreateTablet for tablet 522a91c85b0e4f0d94efaaf5760d5f4f (DEFAULT_TABLE table=test-table [id=9406818d0b5c4a22897accf56add9ebe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:19.447319 10881 raft_consensus.cc:2229] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:19.447402 10749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522a91c85b0e4f0d94efaaf5760d5f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:19.447427 10881 raft_consensus.cc:2258] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:19.447923 10881 ts_tablet_manager.cc:1905] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:19.448273 10609 tablet_service.cc:1467] Processing CreateTablet for tablet 522a91c85b0e4f0d94efaaf5760d5f4f (DEFAULT_TABLE table=test-table [id=9406818d0b5c4a22897accf56add9ebe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:19.448432 10609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522a91c85b0e4f0d94efaaf5760d5f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:19.448949 10880 ts_tablet_manager.cc:1918] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:00:19.449026 10880 log.cc:1196] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/a9d30bf600fa4ba786b9a93e8ee9c4e2
I20260926 15:00:19.449048 10839 tablet_bootstrap.cc:492] T 522a91c85b0e4f0d94efaaf5760d5f4f P 138b3ab43d81457287449c23e0e4124e: Bootstrap starting.
I20260926 15:00:19.449297 10880 ts_tablet_manager.cc:1939] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P 138b3ab43d81457287449c23e0e4124e: Deleting consensus metadata
I20260926 15:00:19.449486 10839 tablet_bootstrap.cc:654] T 522a91c85b0e4f0d94efaaf5760d5f4f P 138b3ab43d81457287449c23e0e4124e: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:19.449940 10317 catalog_manager.cc:4841] TS 138b3ab43d81457287449c23e0e4124e (127.8.102.3:42727): tablet a9d30bf600fa4ba786b9a93e8ee9c4e2 (table test-table [id=03307f867bb64f62b59fe0b152741d82]) successfully deleted
I20260926 15:00:19.450191 10881 ts_tablet_manager.cc:1918] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:00:19.450263 10881 log.cc:1196] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/a9d30bf600fa4ba786b9a93e8ee9c4e2
I20260926 15:00:19.450244 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 522a91c85b0e4f0d94efaaf5760d5f4f (DEFAULT_TABLE table=test-table [id=9406818d0b5c4a22897accf56add9ebe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:19.450352 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522a91c85b0e4f0d94efaaf5760d5f4f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:19.450366 10839 tablet_bootstrap.cc:492] T 522a91c85b0e4f0d94efaaf5760d5f4f P 138b3ab43d81457287449c23e0e4124e: No bootstrap required, opened a new log
I20260926 15:00:19.450421 10839 ts_tablet_manager.cc:1397] T 522a91c85b0e4f0d94efaaf5760d5f4f P 138b3ab43d81457287449c23e0e4124e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:19.450534 10881 ts_tablet_manager.cc:1939] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P d7db7245daa640c588c327d148135a3f: Deleting consensus metadata
I20260926 15:00:19.450572 10839 raft_consensus.cc:348] T 522a91c85b0e4f0d94efaaf5760d5f4f P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:19.450680 10839 raft_consensus.cc:374] T 522a91c85b0e4f0d94efaaf5760d5f4f P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:19.450707 10839 raft_consensus.cc:729] T 522a91c85b0e4f0d94efaaf5760d5f4f P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138b3ab43d81457287449c23e0e4124e, State: Initialized, Role: FOLLOWER
I20260926 15:00:19.450752 10839 consensus_queue.cc:260] T 522a91c85b0e4f0d94efaaf5760d5f4f P 138b3ab43d81457287449c23e0e4124e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:19.450845 10839 ts_tablet_manager.cc:1428] T 522a91c85b0e4f0d94efaaf5760d5f4f P 138b3ab43d81457287449c23e0e4124e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:19.451627 10842 tablet_bootstrap.cc:492] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f: Bootstrap starting.
I20260926 15:00:19.452041 10842 tablet_bootstrap.cc:654] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:19.452041 10879 ts_tablet_manager.cc:1905] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:19.452370 10838 tablet_bootstrap.cc:492] T 522a91c85b0e4f0d94efaaf5760d5f4f P f35cb0916ab84cf8af024aaa83b2c74f: Bootstrap starting.
I20260926 15:00:19.452664 10838 tablet_bootstrap.cc:654] T 522a91c85b0e4f0d94efaaf5760d5f4f P f35cb0916ab84cf8af024aaa83b2c74f: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:19.453117 10842 tablet_bootstrap.cc:492] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f: No bootstrap required, opened a new log
I20260926 15:00:19.453169 10842 ts_tablet_manager.cc:1397] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 15:00:19.453305 10842 raft_consensus.cc:348] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:19.453379 10842 raft_consensus.cc:374] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:19.453398 10842 raft_consensus.cc:729] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7db7245daa640c588c327d148135a3f, State: Initialized, Role: FOLLOWER
I20260926 15:00:19.453442 10842 consensus_queue.cc:260] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:19.453537 10842 ts_tablet_manager.cc:1428] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:19.453696 10879 ts_tablet_manager.cc:1918] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:00:19.453763 10879 log.cc:1196] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-0/wal/wals/a9d30bf600fa4ba786b9a93e8ee9c4e2
I20260926 15:00:19.453881 10838 tablet_bootstrap.cc:492] T 522a91c85b0e4f0d94efaaf5760d5f4f P f35cb0916ab84cf8af024aaa83b2c74f: No bootstrap required, opened a new log
I20260926 15:00:19.453945 10838 ts_tablet_manager.cc:1397] T 522a91c85b0e4f0d94efaaf5760d5f4f P f35cb0916ab84cf8af024aaa83b2c74f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260926 15:00:19.454030 10879 ts_tablet_manager.cc:1939] T a9d30bf600fa4ba786b9a93e8ee9c4e2 P f35cb0916ab84cf8af024aaa83b2c74f: Deleting consensus metadata
I20260926 15:00:19.454109 10838 raft_consensus.cc:348] T 522a91c85b0e4f0d94efaaf5760d5f4f P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:19.454202 10838 raft_consensus.cc:374] T 522a91c85b0e4f0d94efaaf5760d5f4f P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:19.454224 10838 raft_consensus.cc:729] T 522a91c85b0e4f0d94efaaf5760d5f4f P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f35cb0916ab84cf8af024aaa83b2c74f, State: Initialized, Role: FOLLOWER
I20260926 15:00:19.454285 10838 consensus_queue.cc:260] T 522a91c85b0e4f0d94efaaf5760d5f4f P f35cb0916ab84cf8af024aaa83b2c74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:19.454377 10838 ts_tablet_manager.cc:1428] T 522a91c85b0e4f0d94efaaf5760d5f4f P f35cb0916ab84cf8af024aaa83b2c74f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:19.455310 10318 catalog_manager.cc:4841] TS f35cb0916ab84cf8af024aaa83b2c74f (127.8.102.1:42289): tablet a9d30bf600fa4ba786b9a93e8ee9c4e2 (table test-table [id=03307f867bb64f62b59fe0b152741d82]) successfully deleted
I20260926 15:00:19.459165 10319 catalog_manager.cc:4841] TS d7db7245daa640c588c327d148135a3f (127.8.102.2:35893): tablet a9d30bf600fa4ba786b9a93e8ee9c4e2 (table test-table [id=03307f867bb64f62b59fe0b152741d82]) successfully deleted
I20260926 15:00:19.507974 10848 raft_consensus.cc:482] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:19.508081 10848 raft_consensus.cc:504] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:19.508419 10848 leader_election.cc:288] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f35cb0916ab84cf8af024aaa83b2c74f (127.8.102.1:42289), 138b3ab43d81457287449c23e0e4124e (127.8.102.3:42727)
I20260926 15:00:19.512140 10885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.508636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:42727 (local address 127.8.102.2:59705)
0926 15:00:19.508796 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:19.508808 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:19.508861 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:19.509020 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:19.509021 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:19.509023 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:19.509079 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.509081 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.510078 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.510079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.511204 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.511207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.511799 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.511800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.511876 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.511881 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:00:19.511980 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 15:00:19.512032 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"mutex_wait_us":3,"thread_start_us":59,"threads_started":1}
I20260926 15:00:19.512149 10834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.508805 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:59705 (local address 127.8.102.3:42727)
0926 15:00:19.508847 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:19.508849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:19.508858 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:19.508940 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:19.508942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:19.508951 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:19.509000 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:19.509148 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.509973 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.511342 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.511704 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.511797 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.511816 (+ 19us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:19.512019 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 15:00:19.512071 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 15:00:19.512213 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.508674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:39333 (local address 127.8.102.1:42289)
0926 15:00:19.508737 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:19.508740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:19.508860 (+ 120us) server_negotiation.cc:407] Connection header received
0926 15:00:19.508992 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:19.508994 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:19.509005 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:19.509057 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:19.509241 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.509721 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.511313 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.511832 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.511915 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.511928 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:19.512089 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 15:00:19.512139 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 15:00:19.512449 10886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.508601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:42289 (local address 127.8.102.2:39333)
0926 15:00:19.508788 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:19.508801 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:19.508924 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:19.509059 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:19.509060 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:19.509062 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:19.509112 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.509113 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.509828 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.509829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.511204 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.511207 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.511914 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.511915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.511992 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.511995 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:00:19.512327 (+ 332us) client_negotiation.cc:240] Negotiation successful
0926 15:00:19.512376 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"mutex_wait_us":56,"thread_start_us":58,"threads_started":1}
I20260926 15:00:19.512581 10768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522a91c85b0e4f0d94efaaf5760d5f4f" candidate_uuid: "d7db7245daa640c588c327d148135a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138b3ab43d81457287449c23e0e4124e" is_pre_election: true
I20260926 15:00:19.512681 10768 raft_consensus.cc:2454] T 522a91c85b0e4f0d94efaaf5760d5f4f P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7db7245daa640c588c327d148135a3f in term 0.
I20260926 15:00:19.512698 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522a91c85b0e4f0d94efaaf5760d5f4f" candidate_uuid: "d7db7245daa640c588c327d148135a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" is_pre_election: true
I20260926 15:00:19.512843 10489 raft_consensus.cc:2454] T 522a91c85b0e4f0d94efaaf5760d5f4f P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7db7245daa640c588c327d148135a3f in term 0.
I20260926 15:00:19.512900 10561 leader_election.cc:302] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138b3ab43d81457287449c23e0e4124e, d7db7245daa640c588c327d148135a3f; no voters:
I20260926 15:00:19.513067 10848 raft_consensus.cc:2781] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:00:19.513101 10848 raft_consensus.cc:482] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:00:19.513119 10848 raft_consensus.cc:3037] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:19.513847 10848 raft_consensus.cc:504] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:19.514037 10848 leader_election.cc:288] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [CANDIDATE]: Term 1 election: Requested vote from peers f35cb0916ab84cf8af024aaa83b2c74f (127.8.102.1:42289), 138b3ab43d81457287449c23e0e4124e (127.8.102.3:42727)
I20260926 15:00:19.514204 10768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522a91c85b0e4f0d94efaaf5760d5f4f" candidate_uuid: "d7db7245daa640c588c327d148135a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138b3ab43d81457287449c23e0e4124e"
I20260926 15:00:19.514298 10768 raft_consensus.cc:3037] T 522a91c85b0e4f0d94efaaf5760d5f4f P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:19.514294 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522a91c85b0e4f0d94efaaf5760d5f4f" candidate_uuid: "d7db7245daa640c588c327d148135a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f35cb0916ab84cf8af024aaa83b2c74f"
I20260926 15:00:19.514369 10489 raft_consensus.cc:3037] T 522a91c85b0e4f0d94efaaf5760d5f4f P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:19.515199 10489 raft_consensus.cc:2454] T 522a91c85b0e4f0d94efaaf5760d5f4f P f35cb0916ab84cf8af024aaa83b2c74f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7db7245daa640c588c327d148135a3f in term 1.
I20260926 15:00:19.515409 10562 leader_election.cc:302] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7db7245daa640c588c327d148135a3f, f35cb0916ab84cf8af024aaa83b2c74f; no voters:
I20260926 15:00:19.515512 10848 raft_consensus.cc:2781] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:19.515558 10768 raft_consensus.cc:2454] T 522a91c85b0e4f0d94efaaf5760d5f4f P 138b3ab43d81457287449c23e0e4124e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7db7245daa640c588c327d148135a3f in term 1.
I20260926 15:00:19.515626 10848 raft_consensus.cc:686] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [term 1 LEADER]: Becoming Leader. State: Replica: d7db7245daa640c588c327d148135a3f, State: Running, Role: LEADER
I20260926 15:00:19.515703 10848 consensus_queue.cc:237] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:19.516496 10330 catalog_manager.cc:5495] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f reported cstate change: term changed from 0 to 1, leader changed from <none> to d7db7245daa640c588c327d148135a3f (127.8.102.2). New cstate: current_term: 1 leader_uuid: "d7db7245daa640c588c327d148135a3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } health_report { overall_health: UNKNOWN } } }
I20260926 15:00:19.535230 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.532550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49272 (local address 127.8.102.2:35893)
0926 15:00:19.532617 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:19.532619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:19.532626 (+ 7us) server_negotiation.cc:407] Connection header received
0926 15:00:19.532675 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:19.532675 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:19.532698 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:19.532791 (+ 93us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:19.532918 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.533455 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.534297 (+ 842us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.534533 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.534625 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.534802 (+ 177us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:19.534936 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:19.535033 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:19.535084 (+ 51us) server_negotiation.cc:299] Negotiation successful
0926 15:00:19.535110 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 15:00:19.541172 10835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.538009 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:47359 (local address 127.8.102.1:42289)
0926 15:00:19.538060 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:19.538062 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:19.538244 (+ 182us) server_negotiation.cc:407] Connection header received
0926 15:00:19.538304 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:19.538305 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:19.538314 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:19.538368 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:19.538512 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.539273 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.540418 (+ 1145us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.540627 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.540690 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.540921 (+ 231us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:19.540957 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:19.541014 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:19.541092 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 15:00:19.541109 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 15:00:19.541630 10891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.537963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:42289 (local address 127.8.102.2:47359)
0926 15:00:19.538163 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:19.538176 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:19.538242 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:19.538379 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:19.538379 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:19.538381 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:00:19.538454 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.538455 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.539357 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.539358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.540320 (+ 962us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.540322 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.540709 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.540710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.540780 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.540801 (+ 21us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:00:19.541030 (+ 229us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:00:19.541032 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 15:00:19.541086 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 15:00:19.541133 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":97,"threads_started":1}
I20260926 15:00:19.544221 10768 raft_consensus.cc:1261] T 522a91c85b0e4f0d94efaaf5760d5f4f P 138b3ab43d81457287449c23e0e4124e [term 1 FOLLOWER]: Refusing update from remote peer d7db7245daa640c588c327d148135a3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:19.544325 10489 raft_consensus.cc:1261] T 522a91c85b0e4f0d94efaaf5760d5f4f P f35cb0916ab84cf8af024aaa83b2c74f [term 1 FOLLOWER]: Refusing update from remote peer d7db7245daa640c588c327d148135a3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:19.544502 10848 consensus_queue.cc:1035] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:19.544624 10887 consensus_queue.cc:1035] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:19.552531 10874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.549997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:35893 (local address 127.8.102.1:52977)
0926 15:00:19.550062 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:19.550074 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:19.550116 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:19.550333 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:19.550334 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:19.550335 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:00:19.550379 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.550381 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.551074 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.551076 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.551717 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:19.551720 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.552175 (+ 455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.552176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:19.552241 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.552247 (+ 6us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:00:19.552401 (+ 154us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:00:19.552402 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 15:00:19.552432 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 15:00:19.552469 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 15:00:19.552534 10836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:19.550187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:52977 (local address 127.8.102.2:35893)
0926 15:00:19.550232 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:19.550234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:19.550241 (+ 7us) server_negotiation.cc:407] Connection header received
0926 15:00:19.550265 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:19.550266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:19.550275 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:19.550329 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:19.550430 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.550982 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.551796 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:19.552096 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:19.552174 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:19.552312 (+ 138us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:19.552346 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:19.552402 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:19.552436 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 15:00:19.552453 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 15:00:19.552930 10864 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 15:00:21.079135 10330 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:21.079260 10330 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:21.080253 10330 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78: Sending DeleteTablet for 3 replicas of tablet 522a91c85b0e4f0d94efaaf5760d5f4f
I20260926 15:00:21.080533 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet 522a91c85b0e4f0d94efaaf5760d5f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:21 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:59434
I20260926 15:00:21.080586 10609 tablet_service.cc:1514] Processing DeleteTablet for tablet 522a91c85b0e4f0d94efaaf5760d5f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:21 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:49260
I20260926 15:00:21.080610 10749 tablet_service.cc:1514] Processing DeleteTablet for tablet 522a91c85b0e4f0d94efaaf5760d5f4f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:21 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:46754
I20260926 15:00:21.080725 10918 tablet_replica.cc:330] T 522a91c85b0e4f0d94efaaf5760d5f4f P f35cb0916ab84cf8af024aaa83b2c74f: stopping tablet replica
I20260926 15:00:21.080787 10918 raft_consensus.cc:2229] T 522a91c85b0e4f0d94efaaf5760d5f4f P f35cb0916ab84cf8af024aaa83b2c74f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:21.080832 10918 raft_consensus.cc:2258] T 522a91c85b0e4f0d94efaaf5760d5f4f P f35cb0916ab84cf8af024aaa83b2c74f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:21.081017 10919 tablet_replica.cc:330] T 522a91c85b0e4f0d94efaaf5760d5f4f P 138b3ab43d81457287449c23e0e4124e: stopping tablet replica
I20260926 15:00:21.081070 10918 ts_tablet_manager.cc:1905] T 522a91c85b0e4f0d94efaaf5760d5f4f P f35cb0916ab84cf8af024aaa83b2c74f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:21.081085 10919 raft_consensus.cc:2229] T 522a91c85b0e4f0d94efaaf5760d5f4f P 138b3ab43d81457287449c23e0e4124e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:21.081131 10919 raft_consensus.cc:2258] T 522a91c85b0e4f0d94efaaf5760d5f4f P 138b3ab43d81457287449c23e0e4124e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:21.081368 10919 ts_tablet_manager.cc:1905] T 522a91c85b0e4f0d94efaaf5760d5f4f P 138b3ab43d81457287449c23e0e4124e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:21.083024 10918 ts_tablet_manager.cc:1918] T 522a91c85b0e4f0d94efaaf5760d5f4f P f35cb0916ab84cf8af024aaa83b2c74f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 15:00:21.083096 10918 log.cc:1196] T 522a91c85b0e4f0d94efaaf5760d5f4f P f35cb0916ab84cf8af024aaa83b2c74f: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-0/wal/wals/522a91c85b0e4f0d94efaaf5760d5f4f
I20260926 15:00:21.083082 10919 ts_tablet_manager.cc:1918] T 522a91c85b0e4f0d94efaaf5760d5f4f P 138b3ab43d81457287449c23e0e4124e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 15:00:21.083141 10919 log.cc:1196] T 522a91c85b0e4f0d94efaaf5760d5f4f P 138b3ab43d81457287449c23e0e4124e: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/522a91c85b0e4f0d94efaaf5760d5f4f
I20260926 15:00:21.083331 10918 ts_tablet_manager.cc:1939] T 522a91c85b0e4f0d94efaaf5760d5f4f P f35cb0916ab84cf8af024aaa83b2c74f: Deleting consensus metadata
I20260926 15:00:21.083366 10919 ts_tablet_manager.cc:1939] T 522a91c85b0e4f0d94efaaf5760d5f4f P 138b3ab43d81457287449c23e0e4124e: Deleting consensus metadata
I20260926 15:00:21.083526 10920 tablet_replica.cc:330] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f: stopping tablet replica
I20260926 15:00:21.083700 10318 catalog_manager.cc:4841] TS f35cb0916ab84cf8af024aaa83b2c74f (127.8.102.1:42289): tablet 522a91c85b0e4f0d94efaaf5760d5f4f (table test-table [id=9406818d0b5c4a22897accf56add9ebe]) successfully deleted
I20260926 15:00:21.083877 10317 catalog_manager.cc:4841] TS 138b3ab43d81457287449c23e0e4124e (127.8.102.3:42727): tablet 522a91c85b0e4f0d94efaaf5760d5f4f (table test-table [id=9406818d0b5c4a22897accf56add9ebe]) successfully deleted
I20260926 15:00:21.083596 10920 raft_consensus.cc:2229] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:00:21.084056 10920 raft_consensus.cc:2258] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:21.085638 10920 ts_tablet_manager.cc:1905] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:21.087136 10920 ts_tablet_manager.cc:1918] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 15:00:21.087221 10920 log.cc:1196] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/522a91c85b0e4f0d94efaaf5760d5f4f
I20260926 15:00:21.087445 10920 ts_tablet_manager.cc:1939] T 522a91c85b0e4f0d94efaaf5760d5f4f P d7db7245daa640c588c327d148135a3f: Deleting consensus metadata
I20260926 15:00:21.087812 10319 catalog_manager.cc:4841] TS d7db7245daa640c588c327d148135a3f (127.8.102.2:35893): tablet 522a91c85b0e4f0d94efaaf5760d5f4f (table test-table [id=9406818d0b5c4a22897accf56add9ebe]) successfully deleted
Sep 26 15:00:21 dist-test-slave-jnhs krb5kdc[10287](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 15:00:21 dist-test-slave-jnhs krb5kdc[10287](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:21.113672 10928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:21.101171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51518 (local address 127.8.102.62:37383)
0926 15:00:21.101448 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:21.101451 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:21.101460 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:21.101489 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:21.101490 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:21.101501 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:21.101594 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:21.102242 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:21.102883 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:21.103867 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:21.104101 (+ 234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:21.104176 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:21.110807 (+ 6631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:21.110810 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:21.110810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:21.110819 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:21.112403 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:21.112859 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:21.112860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:21.112860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:21.112922 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:21.113097 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:21.113098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:21.113098 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:21.113211 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:21.113291 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:21.113401 (+ 110us) server_negotiation.cc:299] Negotiation successful
0926 15:00:21.113501 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":251,"thread_start_us":176,"threads_started":1}
I20260926 15:00:21.115613 10330 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:00:21.115972 10330 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:00:21.117882 10609 tablet_service.cc:1467] Processing CreateTablet for tablet 541d2c3c684e4d5e8aef900825ea5945 (DEFAULT_TABLE table=test-table [id=1b4f4f16e84b4f1ca8b6085d0b245df6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:21.117920 10469 tablet_service.cc:1467] Processing CreateTablet for tablet 541d2c3c684e4d5e8aef900825ea5945 (DEFAULT_TABLE table=test-table [id=1b4f4f16e84b4f1ca8b6085d0b245df6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:21.117880 10749 tablet_service.cc:1467] Processing CreateTablet for tablet 541d2c3c684e4d5e8aef900825ea5945 (DEFAULT_TABLE table=test-table [id=1b4f4f16e84b4f1ca8b6085d0b245df6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:21.118070 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 541d2c3c684e4d5e8aef900825ea5945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:21.118070 10749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 541d2c3c684e4d5e8aef900825ea5945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:21.118072 10609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 541d2c3c684e4d5e8aef900825ea5945. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:21.119578 10930 tablet_bootstrap.cc:492] T 541d2c3c684e4d5e8aef900825ea5945 P f35cb0916ab84cf8af024aaa83b2c74f: Bootstrap starting.
I20260926 15:00:21.119719 10931 tablet_bootstrap.cc:492] T 541d2c3c684e4d5e8aef900825ea5945 P 138b3ab43d81457287449c23e0e4124e: Bootstrap starting.
I20260926 15:00:21.119721 10929 tablet_bootstrap.cc:492] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f: Bootstrap starting.
I20260926 15:00:21.120093 10929 tablet_bootstrap.cc:654] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:21.120091 10930 tablet_bootstrap.cc:654] T 541d2c3c684e4d5e8aef900825ea5945 P f35cb0916ab84cf8af024aaa83b2c74f: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:21.120209 10931 tablet_bootstrap.cc:654] T 541d2c3c684e4d5e8aef900825ea5945 P 138b3ab43d81457287449c23e0e4124e: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:21.120779 10929 tablet_bootstrap.cc:492] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f: No bootstrap required, opened a new log
I20260926 15:00:21.120836 10929 ts_tablet_manager.cc:1397] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:21.120900 10931 tablet_bootstrap.cc:492] T 541d2c3c684e4d5e8aef900825ea5945 P 138b3ab43d81457287449c23e0e4124e: No bootstrap required, opened a new log
I20260926 15:00:21.120959 10931 ts_tablet_manager.cc:1397] T 541d2c3c684e4d5e8aef900825ea5945 P 138b3ab43d81457287449c23e0e4124e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:21.120959 10929 raft_consensus.cc:348] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:21.121035 10929 raft_consensus.cc:374] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:21.121061 10929 raft_consensus.cc:729] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7db7245daa640c588c327d148135a3f, State: Initialized, Role: FOLLOWER
I20260926 15:00:21.121104 10929 consensus_queue.cc:260] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:21.121217 10929 ts_tablet_manager.cc:1428] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:21.121559 10930 tablet_bootstrap.cc:492] T 541d2c3c684e4d5e8aef900825ea5945 P f35cb0916ab84cf8af024aaa83b2c74f: No bootstrap required, opened a new log
I20260926 15:00:21.121613 10930 ts_tablet_manager.cc:1397] T 541d2c3c684e4d5e8aef900825ea5945 P f35cb0916ab84cf8af024aaa83b2c74f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:00:21.121731 10930 raft_consensus.cc:348] T 541d2c3c684e4d5e8aef900825ea5945 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:21.121812 10930 raft_consensus.cc:374] T 541d2c3c684e4d5e8aef900825ea5945 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:21.121834 10930 raft_consensus.cc:729] T 541d2c3c684e4d5e8aef900825ea5945 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f35cb0916ab84cf8af024aaa83b2c74f, State: Initialized, Role: FOLLOWER
I20260926 15:00:21.121881 10930 consensus_queue.cc:260] T 541d2c3c684e4d5e8aef900825ea5945 P f35cb0916ab84cf8af024aaa83b2c74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:21.121990 10930 ts_tablet_manager.cc:1428] T 541d2c3c684e4d5e8aef900825ea5945 P f35cb0916ab84cf8af024aaa83b2c74f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:21.122326 10931 raft_consensus.cc:348] T 541d2c3c684e4d5e8aef900825ea5945 P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:21.122471 10931 raft_consensus.cc:374] T 541d2c3c684e4d5e8aef900825ea5945 P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:21.122500 10931 raft_consensus.cc:729] T 541d2c3c684e4d5e8aef900825ea5945 P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138b3ab43d81457287449c23e0e4124e, State: Initialized, Role: FOLLOWER
I20260926 15:00:21.122614 10931 consensus_queue.cc:260] T 541d2c3c684e4d5e8aef900825ea5945 P 138b3ab43d81457287449c23e0e4124e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:21.123222 10931 ts_tablet_manager.cc:1428] T 541d2c3c684e4d5e8aef900825ea5945 P 138b3ab43d81457287449c23e0e4124e: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:00:21.353487 10910 raft_consensus.cc:482] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:21.353624 10910 raft_consensus.cc:504] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:21.353844 10910 leader_election.cc:288] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f35cb0916ab84cf8af024aaa83b2c74f (127.8.102.1:42289), 138b3ab43d81457287449c23e0e4124e (127.8.102.3:42727)
I20260926 15:00:21.354077 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "541d2c3c684e4d5e8aef900825ea5945" candidate_uuid: "d7db7245daa640c588c327d148135a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" is_pre_election: true
I20260926 15:00:21.354163 10489 raft_consensus.cc:2454] T 541d2c3c684e4d5e8aef900825ea5945 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7db7245daa640c588c327d148135a3f in term 0.
I20260926 15:00:21.354120 10768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "541d2c3c684e4d5e8aef900825ea5945" candidate_uuid: "d7db7245daa640c588c327d148135a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138b3ab43d81457287449c23e0e4124e" is_pre_election: true
I20260926 15:00:21.354259 10768 raft_consensus.cc:2454] T 541d2c3c684e4d5e8aef900825ea5945 P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7db7245daa640c588c327d148135a3f in term 0.
I20260926 15:00:21.354384 10562 leader_election.cc:302] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7db7245daa640c588c327d148135a3f, f35cb0916ab84cf8af024aaa83b2c74f; no voters:
I20260926 15:00:21.354492 10910 raft_consensus.cc:2781] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:00:21.354547 10910 raft_consensus.cc:482] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:00:21.354604 10910 raft_consensus.cc:3037] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:21.355370 10910 raft_consensus.cc:504] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:21.355531 10910 leader_election.cc:288] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [CANDIDATE]: Term 1 election: Requested vote from peers f35cb0916ab84cf8af024aaa83b2c74f (127.8.102.1:42289), 138b3ab43d81457287449c23e0e4124e (127.8.102.3:42727)
I20260926 15:00:21.355695 10489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "541d2c3c684e4d5e8aef900825ea5945" candidate_uuid: "d7db7245daa640c588c327d148135a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f35cb0916ab84cf8af024aaa83b2c74f"
I20260926 15:00:21.355814 10489 raft_consensus.cc:3037] T 541d2c3c684e4d5e8aef900825ea5945 P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:21.355800 10768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "541d2c3c684e4d5e8aef900825ea5945" candidate_uuid: "d7db7245daa640c588c327d148135a3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138b3ab43d81457287449c23e0e4124e"
I20260926 15:00:21.355890 10768 raft_consensus.cc:3037] T 541d2c3c684e4d5e8aef900825ea5945 P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:21.356498 10768 raft_consensus.cc:2454] T 541d2c3c684e4d5e8aef900825ea5945 P 138b3ab43d81457287449c23e0e4124e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7db7245daa640c588c327d148135a3f in term 1.
I20260926 15:00:21.356568 10489 raft_consensus.cc:2454] T 541d2c3c684e4d5e8aef900825ea5945 P f35cb0916ab84cf8af024aaa83b2c74f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7db7245daa640c588c327d148135a3f in term 1.
I20260926 15:00:21.356705 10561 leader_election.cc:302] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138b3ab43d81457287449c23e0e4124e, d7db7245daa640c588c327d148135a3f; no voters:
I20260926 15:00:21.356804 10910 raft_consensus.cc:2781] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:21.356868 10910 raft_consensus.cc:686] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [term 1 LEADER]: Becoming Leader. State: Replica: d7db7245daa640c588c327d148135a3f, State: Running, Role: LEADER
I20260926 15:00:21.356935 10910 consensus_queue.cc:237] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:21.357653 10330 catalog_manager.cc:5495] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f reported cstate change: term changed from 0 to 1, leader changed from <none> to d7db7245daa640c588c327d148135a3f (127.8.102.2). New cstate: current_term: 1 leader_uuid: "d7db7245daa640c588c327d148135a3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } health_report { overall_health: UNKNOWN } } }
I20260926 15:00:21.410281 10938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:21.407730 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49288 (local address 127.8.102.2:35893)
0926 15:00:21.407901 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:21.407903 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:21.407912 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:21.407937 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:21.407937 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:21.407947 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:21.408026 (+ 79us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:21.408174 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:21.408676 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:21.409452 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:21.409728 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:21.409811 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:21.409983 (+ 172us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:21.410027 (+ 44us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:21.410089 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:21.410152 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 15:00:21.410178 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260926 15:00:21.412719 10768 raft_consensus.cc:1261] T 541d2c3c684e4d5e8aef900825ea5945 P 138b3ab43d81457287449c23e0e4124e [term 1 FOLLOWER]: Refusing update from remote peer d7db7245daa640c588c327d148135a3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:21.412869 10489 raft_consensus.cc:1261] T 541d2c3c684e4d5e8aef900825ea5945 P f35cb0916ab84cf8af024aaa83b2c74f [term 1 FOLLOWER]: Refusing update from remote peer d7db7245daa640c588c327d148135a3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:21.412974 10910 consensus_queue.cc:1035] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:21.413110 10896 consensus_queue.cc:1035] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:21.423167 10331 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:21.423269 10331 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:21.423951 10331 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78: Sending DeleteTablet for 3 replicas of tablet 541d2c3c684e4d5e8aef900825ea5945
I20260926 15:00:21.424255 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet 541d2c3c684e4d5e8aef900825ea5945 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:21 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:59434
I20260926 15:00:21.424301 10749 tablet_service.cc:1514] Processing DeleteTablet for tablet 541d2c3c684e4d5e8aef900825ea5945 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:21 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:46754
I20260926 15:00:21.424394 10918 tablet_replica.cc:330] T 541d2c3c684e4d5e8aef900825ea5945 P f35cb0916ab84cf8af024aaa83b2c74f: stopping tablet replica
I20260926 15:00:21.424400 10919 tablet_replica.cc:330] T 541d2c3c684e4d5e8aef900825ea5945 P 138b3ab43d81457287449c23e0e4124e: stopping tablet replica
I20260926 15:00:21.424455 10919 raft_consensus.cc:2229] T 541d2c3c684e4d5e8aef900825ea5945 P 138b3ab43d81457287449c23e0e4124e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:21.424458 10918 raft_consensus.cc:2229] T 541d2c3c684e4d5e8aef900825ea5945 P f35cb0916ab84cf8af024aaa83b2c74f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:21.424502 10918 raft_consensus.cc:2258] T 541d2c3c684e4d5e8aef900825ea5945 P f35cb0916ab84cf8af024aaa83b2c74f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:21.424520 10919 raft_consensus.cc:2258] T 541d2c3c684e4d5e8aef900825ea5945 P 138b3ab43d81457287449c23e0e4124e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:21.424657 10609 tablet_service.cc:1514] Processing DeleteTablet for tablet 541d2c3c684e4d5e8aef900825ea5945 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:21 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:49260
I20260926 15:00:21.424600 10331 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260926 15:00:21.424751 10920 tablet_replica.cc:330] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f: stopping tablet replica
W20260926 15:00:21.424757 10331 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:00:21.424808 10920 raft_consensus.cc:2229] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:00:21.424871 10920 raft_consensus.cc:2258] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:21.424949 10919 ts_tablet_manager.cc:1905] T 541d2c3c684e4d5e8aef900825ea5945 P 138b3ab43d81457287449c23e0e4124e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:21.425314 10918 ts_tablet_manager.cc:1905] T 541d2c3c684e4d5e8aef900825ea5945 P f35cb0916ab84cf8af024aaa83b2c74f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:21.425417 10920 ts_tablet_manager.cc:1905] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:21.426951 10919 ts_tablet_manager.cc:1918] T 541d2c3c684e4d5e8aef900825ea5945 P 138b3ab43d81457287449c23e0e4124e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:00:21.427066 10919 log.cc:1196] T 541d2c3c684e4d5e8aef900825ea5945 P 138b3ab43d81457287449c23e0e4124e: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/541d2c3c684e4d5e8aef900825ea5945
I20260926 15:00:21.427256 10920 ts_tablet_manager.cc:1918] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:00:21.427301 10919 ts_tablet_manager.cc:1939] T 541d2c3c684e4d5e8aef900825ea5945 P 138b3ab43d81457287449c23e0e4124e: Deleting consensus metadata
I20260926 15:00:21.427323 10920 log.cc:1196] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/541d2c3c684e4d5e8aef900825ea5945
I20260926 15:00:21.427567 10920 ts_tablet_manager.cc:1939] T 541d2c3c684e4d5e8aef900825ea5945 P d7db7245daa640c588c327d148135a3f: Deleting consensus metadata
I20260926 15:00:21.427649 10317 catalog_manager.cc:4841] TS 138b3ab43d81457287449c23e0e4124e (127.8.102.3:42727): tablet 541d2c3c684e4d5e8aef900825ea5945 (table test-table [id=1b4f4f16e84b4f1ca8b6085d0b245df6]) successfully deleted
I20260926 15:00:21.427913 10918 ts_tablet_manager.cc:1918] T 541d2c3c684e4d5e8aef900825ea5945 P f35cb0916ab84cf8af024aaa83b2c74f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:00:21.427980 10918 log.cc:1196] T 541d2c3c684e4d5e8aef900825ea5945 P f35cb0916ab84cf8af024aaa83b2c74f: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-0/wal/wals/541d2c3c684e4d5e8aef900825ea5945
I20260926 15:00:21.428016 10319 catalog_manager.cc:4841] TS d7db7245daa640c588c327d148135a3f (127.8.102.2:35893): tablet 541d2c3c684e4d5e8aef900825ea5945 (table test-table [id=1b4f4f16e84b4f1ca8b6085d0b245df6]) successfully deleted
I20260926 15:00:21.428236 10918 ts_tablet_manager.cc:1939] T 541d2c3c684e4d5e8aef900825ea5945 P f35cb0916ab84cf8af024aaa83b2c74f: Deleting consensus metadata
I20260926 15:00:21.428269 10749 tablet_service.cc:1467] Processing CreateTablet for tablet fa8bf4fc7af4429e957a6d8a7e7f773d (DEFAULT_TABLE table=test-table [id=96179db183ba46d8bb8132f9cd8b537a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:21.428406 10749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8bf4fc7af4429e957a6d8a7e7f773d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:21.428619 10318 catalog_manager.cc:4841] TS f35cb0916ab84cf8af024aaa83b2c74f (127.8.102.1:42289): tablet 541d2c3c684e4d5e8aef900825ea5945 (table test-table [id=1b4f4f16e84b4f1ca8b6085d0b245df6]) successfully deleted
I20260926 15:00:21.428706 10469 tablet_service.cc:1467] Processing CreateTablet for tablet fa8bf4fc7af4429e957a6d8a7e7f773d (DEFAULT_TABLE table=test-table [id=96179db183ba46d8bb8132f9cd8b537a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:21.428755 10609 tablet_service.cc:1467] Processing CreateTablet for tablet fa8bf4fc7af4429e957a6d8a7e7f773d (DEFAULT_TABLE table=test-table [id=96179db183ba46d8bb8132f9cd8b537a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:21.428814 10469 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8bf4fc7af4429e957a6d8a7e7f773d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:21.428856 10609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fa8bf4fc7af4429e957a6d8a7e7f773d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:21.429677 10931 tablet_bootstrap.cc:492] T fa8bf4fc7af4429e957a6d8a7e7f773d P 138b3ab43d81457287449c23e0e4124e: Bootstrap starting.
I20260926 15:00:21.429963 10931 tablet_bootstrap.cc:654] T fa8bf4fc7af4429e957a6d8a7e7f773d P 138b3ab43d81457287449c23e0e4124e: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:21.430122 10929 tablet_bootstrap.cc:492] T fa8bf4fc7af4429e957a6d8a7e7f773d P d7db7245daa640c588c327d148135a3f: Bootstrap starting.
I20260926 15:00:21.430316 10930 tablet_bootstrap.cc:492] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f: Bootstrap starting.
I20260926 15:00:21.430477 10929 tablet_bootstrap.cc:654] T fa8bf4fc7af4429e957a6d8a7e7f773d P d7db7245daa640c588c327d148135a3f: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:21.430685 10931 tablet_bootstrap.cc:492] T fa8bf4fc7af4429e957a6d8a7e7f773d P 138b3ab43d81457287449c23e0e4124e: No bootstrap required, opened a new log
I20260926 15:00:21.430701 10930 tablet_bootstrap.cc:654] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:21.430733 10931 ts_tablet_manager.cc:1397] T fa8bf4fc7af4429e957a6d8a7e7f773d P 138b3ab43d81457287449c23e0e4124e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:21.430888 10931 raft_consensus.cc:348] T fa8bf4fc7af4429e957a6d8a7e7f773d P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:21.430979 10931 raft_consensus.cc:374] T fa8bf4fc7af4429e957a6d8a7e7f773d P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:21.431001 10931 raft_consensus.cc:729] T fa8bf4fc7af4429e957a6d8a7e7f773d P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 138b3ab43d81457287449c23e0e4124e, State: Initialized, Role: FOLLOWER
I20260926 15:00:21.431052 10931 consensus_queue.cc:260] T fa8bf4fc7af4429e957a6d8a7e7f773d P 138b3ab43d81457287449c23e0e4124e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:21.431133 10929 tablet_bootstrap.cc:492] T fa8bf4fc7af4429e957a6d8a7e7f773d P d7db7245daa640c588c327d148135a3f: No bootstrap required, opened a new log
I20260926 15:00:21.431151 10931 ts_tablet_manager.cc:1428] T fa8bf4fc7af4429e957a6d8a7e7f773d P 138b3ab43d81457287449c23e0e4124e: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:21.431183 10929 ts_tablet_manager.cc:1397] T fa8bf4fc7af4429e957a6d8a7e7f773d P d7db7245daa640c588c327d148135a3f: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:21.431322 10929 raft_consensus.cc:348] T fa8bf4fc7af4429e957a6d8a7e7f773d P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:21.431407 10929 raft_consensus.cc:374] T fa8bf4fc7af4429e957a6d8a7e7f773d P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:21.431432 10929 raft_consensus.cc:729] T fa8bf4fc7af4429e957a6d8a7e7f773d P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7db7245daa640c588c327d148135a3f, State: Initialized, Role: FOLLOWER
I20260926 15:00:21.431479 10929 consensus_queue.cc:260] T fa8bf4fc7af4429e957a6d8a7e7f773d P d7db7245daa640c588c327d148135a3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:21.431591 10929 ts_tablet_manager.cc:1428] T fa8bf4fc7af4429e957a6d8a7e7f773d P d7db7245daa640c588c327d148135a3f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:21.431943 10930 tablet_bootstrap.cc:492] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f: No bootstrap required, opened a new log
I20260926 15:00:21.432015 10930 ts_tablet_manager.cc:1397] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 15:00:21.432138 10930 raft_consensus.cc:348] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:21.432214 10930 raft_consensus.cc:374] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:21.432236 10930 raft_consensus.cc:729] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f35cb0916ab84cf8af024aaa83b2c74f, State: Initialized, Role: FOLLOWER
I20260926 15:00:21.432284 10930 consensus_queue.cc:260] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:21.432376 10930 ts_tablet_manager.cc:1428] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:21.621155 10905 raft_consensus.cc:482] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:21.621258 10905 raft_consensus.cc:504] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:21.621421 10905 leader_election.cc:288] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7db7245daa640c588c327d148135a3f (127.8.102.2:35893), 138b3ab43d81457287449c23e0e4124e (127.8.102.3:42727)
I20260926 15:00:21.621646 10768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8bf4fc7af4429e957a6d8a7e7f773d" candidate_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138b3ab43d81457287449c23e0e4124e" is_pre_election: true
I20260926 15:00:21.621646 10629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8bf4fc7af4429e957a6d8a7e7f773d" candidate_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7db7245daa640c588c327d148135a3f" is_pre_election: true
I20260926 15:00:21.621757 10629 raft_consensus.cc:2454] T fa8bf4fc7af4429e957a6d8a7e7f773d P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f35cb0916ab84cf8af024aaa83b2c74f in term 0.
I20260926 15:00:21.621757 10768 raft_consensus.cc:2454] T fa8bf4fc7af4429e957a6d8a7e7f773d P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f35cb0916ab84cf8af024aaa83b2c74f in term 0.
I20260926 15:00:21.621934 10421 leader_election.cc:302] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 138b3ab43d81457287449c23e0e4124e, f35cb0916ab84cf8af024aaa83b2c74f; no voters:
I20260926 15:00:21.622059 10905 raft_consensus.cc:2781] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:00:21.622104 10905 raft_consensus.cc:482] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:00:21.622118 10905 raft_consensus.cc:3037] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:21.622853 10905 raft_consensus.cc:504] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:21.622990 10905 leader_election.cc:288] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f [CANDIDATE]: Term 1 election: Requested vote from peers d7db7245daa640c588c327d148135a3f (127.8.102.2:35893), 138b3ab43d81457287449c23e0e4124e (127.8.102.3:42727)
I20260926 15:00:21.623126 10629 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8bf4fc7af4429e957a6d8a7e7f773d" candidate_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7db7245daa640c588c327d148135a3f"
I20260926 15:00:21.623193 10768 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fa8bf4fc7af4429e957a6d8a7e7f773d" candidate_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "138b3ab43d81457287449c23e0e4124e"
I20260926 15:00:21.623215 10629 raft_consensus.cc:3037] T fa8bf4fc7af4429e957a6d8a7e7f773d P d7db7245daa640c588c327d148135a3f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:21.623246 10768 raft_consensus.cc:3037] T fa8bf4fc7af4429e957a6d8a7e7f773d P 138b3ab43d81457287449c23e0e4124e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:21.623910 10629 raft_consensus.cc:2454] T fa8bf4fc7af4429e957a6d8a7e7f773d P d7db7245daa640c588c327d148135a3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f35cb0916ab84cf8af024aaa83b2c74f in term 1.
I20260926 15:00:21.623973 10768 raft_consensus.cc:2454] T fa8bf4fc7af4429e957a6d8a7e7f773d P 138b3ab43d81457287449c23e0e4124e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f35cb0916ab84cf8af024aaa83b2c74f in term 1.
I20260926 15:00:21.624133 10423 leader_election.cc:302] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d7db7245daa640c588c327d148135a3f, f35cb0916ab84cf8af024aaa83b2c74f; no voters:
I20260926 15:00:21.624226 10905 raft_consensus.cc:2781] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:21.624279 10905 raft_consensus.cc:686] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f [term 1 LEADER]: Becoming Leader. State: Replica: f35cb0916ab84cf8af024aaa83b2c74f, State: Running, Role: LEADER
I20260926 15:00:21.624325 10905 consensus_queue.cc:237] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } }
I20260926 15:00:21.624948 10330 catalog_manager.cc:5495] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f reported cstate change: term changed from 0 to 1, leader changed from <none> to f35cb0916ab84cf8af024aaa83b2c74f (127.8.102.1). New cstate: current_term: 1 leader_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f35cb0916ab84cf8af024aaa83b2c74f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 42289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 } health_report { overall_health: UNKNOWN } } }
I20260926 15:00:21.662391 10947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:21.659926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59474 (local address 127.8.102.1:42289)
0926 15:00:21.660141 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:21.660143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:21.660151 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:00:21.660173 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:21.660174 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:21.660184 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:21.660250 (+ 66us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:21.660365 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:21.660876 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:21.661657 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:21.661854 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:21.661916 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:21.662064 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:21.662095 (+ 31us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:21.662161 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:21.662243 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 15:00:21.662272 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":120,"threads_started":1}
I20260926 15:00:21.664613 10768 raft_consensus.cc:1261] T fa8bf4fc7af4429e957a6d8a7e7f773d P 138b3ab43d81457287449c23e0e4124e [term 1 FOLLOWER]: Refusing update from remote peer f35cb0916ab84cf8af024aaa83b2c74f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:21.664650 10629 raft_consensus.cc:1261] T fa8bf4fc7af4429e957a6d8a7e7f773d P d7db7245daa640c588c327d148135a3f [term 1 FOLLOWER]: Refusing update from remote peer f35cb0916ab84cf8af024aaa83b2c74f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:21.664845 10897 consensus_queue.cc:1035] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f [LEADER]: Connected to new peer: Peer: permanent_uuid: "138b3ab43d81457287449c23e0e4124e" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42727 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:21.664942 10900 consensus_queue.cc:1035] T fa8bf4fc7af4429e957a6d8a7e7f773d P f35cb0916ab84cf8af024aaa83b2c74f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7db7245daa640c588c327d148135a3f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 15:00:21.669364 10898 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260926 15:00:23.188375 10332 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:23.188529 10332 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:51518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:23.189472 10332 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8050e2da320f4e5c95099067a5064b78: Sending DeleteTablet for 3 replicas of tablet fa8bf4fc7af4429e957a6d8a7e7f773d
I20260926 15:00:23.189802 10469 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8bf4fc7af4429e957a6d8a7e7f773d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:23 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:59434
I20260926 15:00:23.189815 10749 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8bf4fc7af4429e957a6d8a7e7f773d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:23 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:46754
I20260926 15:00:23.189960 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 10404
I20260926 15:00:23.189976 10609 tablet_service.cc:1514] Processing DeleteTablet for tablet fa8bf4fc7af4429e957a6d8a7e7f773d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:23 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:49260
I20260926 15:00:23.191144 10973 tablet_replica.cc:330] T fa8bf4fc7af4429e957a6d8a7e7f773d P d7db7245daa640c588c327d148135a3f: stopping tablet replica
I20260926 15:00:23.191221 10973 raft_consensus.cc:2229] T fa8bf4fc7af4429e957a6d8a7e7f773d P d7db7245daa640c588c327d148135a3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:23.191272 10973 raft_consensus.cc:2258] T fa8bf4fc7af4429e957a6d8a7e7f773d P d7db7245daa640c588c327d148135a3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:23.191525 10973 ts_tablet_manager.cc:1905] T fa8bf4fc7af4429e957a6d8a7e7f773d P d7db7245daa640c588c327d148135a3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:23.193629 10973 ts_tablet_manager.cc:1918] T fa8bf4fc7af4429e957a6d8a7e7f773d P d7db7245daa640c588c327d148135a3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 15:00:23.193702 10973 log.cc:1196] T fa8bf4fc7af4429e957a6d8a7e7f773d P d7db7245daa640c588c327d148135a3f: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/fa8bf4fc7af4429e957a6d8a7e7f773d
I20260926 15:00:23.193921 10973 ts_tablet_manager.cc:1939] T fa8bf4fc7af4429e957a6d8a7e7f773d P d7db7245daa640c588c327d148135a3f: Deleting consensus metadata
I20260926 15:00:23.194317 10319 catalog_manager.cc:4841] TS d7db7245daa640c588c327d148135a3f (127.8.102.2:35893): tablet fa8bf4fc7af4429e957a6d8a7e7f773d (table test-table [id=96179db183ba46d8bb8132f9cd8b537a]) successfully deleted
I20260926 15:00:23.194777 10972 tablet_replica.cc:330] T fa8bf4fc7af4429e957a6d8a7e7f773d P 138b3ab43d81457287449c23e0e4124e: stopping tablet replica
I20260926 15:00:23.194873 10972 raft_consensus.cc:2229] T fa8bf4fc7af4429e957a6d8a7e7f773d P 138b3ab43d81457287449c23e0e4124e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:23.194955 10972 raft_consensus.cc:2258] T fa8bf4fc7af4429e957a6d8a7e7f773d P 138b3ab43d81457287449c23e0e4124e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:23.195199 10972 ts_tablet_manager.cc:1905] T fa8bf4fc7af4429e957a6d8a7e7f773d P 138b3ab43d81457287449c23e0e4124e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:23.196631 10972 ts_tablet_manager.cc:1918] T fa8bf4fc7af4429e957a6d8a7e7f773d P 138b3ab43d81457287449c23e0e4124e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260926 15:00:23.196722 10972 log.cc:1196] T fa8bf4fc7af4429e957a6d8a7e7f773d P 138b3ab43d81457287449c23e0e4124e: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/fa8bf4fc7af4429e957a6d8a7e7f773d
I20260926 15:00:23.196969 10972 ts_tablet_manager.cc:1939] T fa8bf4fc7af4429e957a6d8a7e7f773d P 138b3ab43d81457287449c23e0e4124e: Deleting consensus metadata
I20260926 15:00:23.197402 10317 catalog_manager.cc:4841] TS 138b3ab43d81457287449c23e0e4124e (127.8.102.3:42727): tablet fa8bf4fc7af4429e957a6d8a7e7f773d (table test-table [id=96179db183ba46d8bb8132f9cd8b537a]) successfully deleted
W20260926 15:00:23.198623 10318 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.102.1:42289 (error 108)
I20260926 15:00:23.198791 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 10544
I20260926 15:00:23.200022 10975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:23.199766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:42289 (local address 127.0.0.1:33690)
0926 15:00:23.199919 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:23.199969 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.1:42289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":68,"threads_started":1}
W20260926 15:00:23.200150 10318 catalog_manager.cc:4584] TS f35cb0916ab84cf8af024aaa83b2c74f (127.8.102.1:42289): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet fa8bf4fc7af4429e957a6d8a7e7f773d: Network error: Client connection negotiation failed: client connection to 127.8.102.1:42289: connect: Connection refused (error 111)
I20260926 15:00:23.204485 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 10684
I20260926 15:00:23.209563 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 10300
2026-09-26T15:00:23Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5935 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:00:23 dist-test-slave-jnhs krb5kdc[10978](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:00:23 dist-test-slave-jnhs krb5kdc[10978](info): set up 2 sockets
Sep 26 15:00:23 dist-test-slave-jnhs krb5kdc[10978](info): commencing operation
krb5kdc: starting...
W20260926 15:00:24.319096 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.087s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:00:24 dist-test-slave-jnhs krb5kdc[10978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:00:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:00:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:00:24.450078 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:40645
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:40645
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:24.533866 10991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:24.534128 10991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:24.534170 10991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:24.535372 10991 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:24.535450 10991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:24.535472 10991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:24.535494 10991 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:24.535521 10991 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:00:24.537631 10991 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:40645
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:40645
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.10991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:24.537930 10991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:24.538189 10991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 15:00:24.544097 10991 server_base.cc:1034] running on GCE node
W20260926 15:00:24.544059 10997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:24.544066 10996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:24.544059 10999 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:24.544723 10991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:24.545090 10991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:24.546276 10991 hybrid_clock.cc:648] HybridClock initialized: now 1790434824546265 us; error 34 us; skew 500 ppm
Sep 26 15:00:24 dist-test-slave-jnhs krb5kdc[10978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434824, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:24.549125 10991 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:00:24.549643 10991 webserver.cc:460] Webserver started at http://127.8.102.62:40697/ using document root <none> and password file <none>
I20260926 15:00:24.549906 10991 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:24.549973 10991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:24.550127 10991 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:24.551277 10991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "d808ac689c95458a8b89292bd7859e7a"
format_stamp: "Formatted at 2026-09-26 15:00:24 on dist-test-slave-jnhs"
server_key: "d7d48e29e9cc305c6fdff2d72602da65"
server_key_iv: "61fa1a171df64f9105a617ba6fe551a2"
server_key_version: "encryptionkey@0"
I20260926 15:00:24.551654 10991 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "d808ac689c95458a8b89292bd7859e7a"
format_stamp: "Formatted at 2026-09-26 15:00:24 on dist-test-slave-jnhs"
server_key: "d7d48e29e9cc305c6fdff2d72602da65"
server_key_iv: "61fa1a171df64f9105a617ba6fe551a2"
server_key_version: "encryptionkey@0"
I20260926 15:00:24.553126 10991 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:00:24.553920 11006 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:24.554121 10991 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:24.554245 10991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "d808ac689c95458a8b89292bd7859e7a"
format_stamp: "Formatted at 2026-09-26 15:00:24 on dist-test-slave-jnhs"
server_key: "d7d48e29e9cc305c6fdff2d72602da65"
server_key_iv: "61fa1a171df64f9105a617ba6fe551a2"
server_key_version: "encryptionkey@0"
I20260926 15:00:24.554327 10991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:24.564939 10991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:24.566967 10991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:24.567145 10991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:24.571319 10991 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:40645
I20260926 15:00:24.571357 11058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:40645 every 8 connection(s)
I20260926 15:00:24.571722 10991 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:00:24.572656 11059 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:24.575349 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 10991
I20260926 15:00:24.575342 11059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a: Bootstrap starting.
I20260926 15:00:24.575451 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:00:24.575629 8600 external_mini_cluster.cc:1428] Setting key fdfea403c3e61a7645f5d8fd0c28f04f
I20260926 15:00:24.576143 11059 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:24.576500 11059 log.cc:824] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:24.577330 11059 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a: No bootstrap required, opened a new log
I20260926 15:00:24.579808 11059 raft_consensus.cc:348] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d808ac689c95458a8b89292bd7859e7a" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 40645 } }
I20260926 15:00:24.579988 11059 raft_consensus.cc:374] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:24.580020 11059 raft_consensus.cc:729] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d808ac689c95458a8b89292bd7859e7a, State: Initialized, Role: FOLLOWER
I20260926 15:00:24.580144 11059 consensus_queue.cc:260] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d808ac689c95458a8b89292bd7859e7a" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 40645 } }
I20260926 15:00:24.580240 11059 raft_consensus.cc:388] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:00:24.580276 11059 raft_consensus.cc:482] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:00:24.580325 11059 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:24.580992 11059 raft_consensus.cc:504] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d808ac689c95458a8b89292bd7859e7a" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 40645 } }
I20260926 15:00:24.581115 11059 leader_election.cc:302] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d808ac689c95458a8b89292bd7859e7a; no voters:
I20260926 15:00:24.581295 11059 leader_election.cc:288] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:00:24.581342 11064 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:24.581538 11064 raft_consensus.cc:686] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a [term 1 LEADER]: Becoming Leader. State: Replica: d808ac689c95458a8b89292bd7859e7a, State: Running, Role: LEADER
I20260926 15:00:24.581609 11059 sys_catalog.cc:564] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:24.581737 11064 consensus_queue.cc:237] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d808ac689c95458a8b89292bd7859e7a" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 40645 } }
I20260926 15:00:24.582197 11066 sys_catalog.cc:455] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a [sys.catalog]: SysCatalogTable state changed. Reason: New leader d808ac689c95458a8b89292bd7859e7a. Latest consensus state: current_term: 1 leader_uuid: "d808ac689c95458a8b89292bd7859e7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d808ac689c95458a8b89292bd7859e7a" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 40645 } } }
I20260926 15:00:24.582296 11066 sys_catalog.cc:458] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:24.582203 11065 sys_catalog.cc:455] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d808ac689c95458a8b89292bd7859e7a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d808ac689c95458a8b89292bd7859e7a" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 40645 } } }
I20260926 15:00:24.582472 11065 sys_catalog.cc:458] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:24.582738 11071 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:00:24.583292 11071 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 15:00:24 dist-test-slave-jnhs krb5kdc[10978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434824, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:24.585310 11071 catalog_manager.cc:1347] Generated new cluster ID: 7503890d9d264b5d8409ddd0db360447
I20260926 15:00:24.585381 11071 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:00:24.587857 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:24.576203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.8.102.62:40645)
0926 15:00:24.576426 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:24.576441 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:24.576468 (+ 27us) server_negotiation.cc:407] Connection header received
0926 15:00:24.577098 (+ 630us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:24.577106 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:24.577161 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:24.577287 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:24.578131 (+ 844us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:24.579151 (+ 1020us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:24.580251 (+ 1100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:24.580792 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:24.580893 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:24.584021 (+ 3128us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:24.584026 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:24.584026 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:24.584039 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:24.586135 (+ 2096us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:24.586582 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:24.586584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:24.586585 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:24.586711 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:24.586923 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:24.586924 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:24.586924 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:24.587222 (+ 298us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:24.587369 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:24.587513 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 15:00:24.587643 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":112,"threads_started":1}
I20260926 15:00:24.607025 11071 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:00:24.607633 11071 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:00:24.612162 11071 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d808ac689c95458a8b89292bd7859e7a: Generated new TSK 0
I20260926 15:00:24.612402 11071 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
I20260926 15:00:24.650102 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:40645
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:24.758240 11085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:24.758458 11085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:24.758482 11085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:24.759778 11085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:24.759829 11085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:24.759859 11085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:00:24.761355 11085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:40645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:24.761554 11085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:24.761739 11085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:24.767001 11090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:24.767000 11091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:24.767041 11093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:24.767342 11085 server_base.cc:1034] running on GCE node
I20260926 15:00:24.767587 11085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:24.767818 11085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:24.768956 11085 hybrid_clock.cc:648] HybridClock initialized: now 1790434824768942 us; error 30 us; skew 500 ppm
Sep 26 15:00:24 dist-test-slave-jnhs krb5kdc[10978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434824, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:24.771798 11085 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:00:24.772300 11085 webserver.cc:460] Webserver started at http://127.8.102.1:38417/ using document root <none> and password file <none>
I20260926 15:00:24.772570 11085 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:24.772650 11085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:24.772814 11085 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:24.774015 11085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "d03add99d1414186a862374f49f90a8e"
format_stamp: "Formatted at 2026-09-26 15:00:24 on dist-test-slave-jnhs"
server_key: "2d902daf3b6659d1d43ec6356b5b182b"
server_key_iv: "4344dba442e63551d5fc1620ca1305d3"
server_key_version: "encryptionkey@0"
I20260926 15:00:24.774420 11085 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "d03add99d1414186a862374f49f90a8e"
format_stamp: "Formatted at 2026-09-26 15:00:24 on dist-test-slave-jnhs"
server_key: "2d902daf3b6659d1d43ec6356b5b182b"
server_key_iv: "4344dba442e63551d5fc1620ca1305d3"
server_key_version: "encryptionkey@0"
I20260926 15:00:24.776011 11085 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:24.777002 11100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:24.777271 11085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:24.777381 11085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "d03add99d1414186a862374f49f90a8e"
format_stamp: "Formatted at 2026-09-26 15:00:24 on dist-test-slave-jnhs"
server_key: "2d902daf3b6659d1d43ec6356b5b182b"
server_key_iv: "4344dba442e63551d5fc1620ca1305d3"
server_key_version: "encryptionkey@0"
I20260926 15:00:24.777468 11085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:24.789395 11085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:24.791885 11085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:24.792088 11085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:24.792369 11085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:24.792768 11085 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:24.792832 11085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:24.792882 11085 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:24.792912 11085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:24.799235 11085 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:32823
I20260926 15:00:24.799307 11213 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:32823 every 8 connection(s)
I20260926 15:00:24.799677 11085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:00:24.801843 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 11085
I20260926 15:00:24.801931 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:00:24.802140 8600 external_mini_cluster.cc:1428] Setting key 07ba0785114c73fbfe14ec1f41713201
Sep 26 15:00:24 dist-test-slave-jnhs krb5kdc[10978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434824, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.102.2@KRBTEST.COM; defaulting to no policy
I20260926 15:00:24.810175 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:24.800473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:48907 (local address 127.8.102.62:40645)
0926 15:00:24.800546 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:24.800548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:24.800805 (+ 257us) server_negotiation.cc:407] Connection header received
0926 15:00:24.801337 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:24.801338 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:24.801350 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:24.801403 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:24.802552 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:24.803221 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:24.804525 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:24.804743 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:24.804808 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:24.807247 (+ 2439us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:24.807249 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:24.807249 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:24.807258 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:24.808691 (+ 1433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:24.809143 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:24.809145 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:24.809146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:24.809209 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:24.809465 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:24.809467 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:24.809467 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:24.809624 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:24.809688 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:24.809971 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 15:00:24.810057 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 15:00:24.810385 11216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:24.800450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:40645 (local address 127.8.102.1:48907)
0926 15:00:24.800675 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:24.800712 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:24.801228 (+ 516us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:24.801491 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:24.801495 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:24.802144 (+ 649us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:24.802459 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:24.802465 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:24.803329 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:24.803330 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:24.804425 (+ 1095us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:24.804429 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:24.804840 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:24.804842 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:24.804999 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:24.805617 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:24.805626 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:24.807128 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:24.808786 (+ 1658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:24.808793 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:24.808796 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:24.809041 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:24.809278 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:24.809279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:24.809280 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:24.809395 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:24.809681 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:24.809688 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:24.809836 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 15:00:24.809951 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 15:00:24.810125 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":98,"threads_started":1}
I20260926 15:00:24.811034 11214 heartbeater.cc:347] Connected to a master server at 127.8.102.62:40645
I20260926 15:00:24.811187 11214 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:24.811388 11214 heartbeater.cc:511] Master 127.8.102.62:40645 requested a full tablet report, sending...
I20260926 15:00:24.811985 11023 ts_manager.cc:194] Registered new tserver with Master: d03add99d1414186a862374f49f90a8e (127.8.102.1:32823)
I20260926 15:00:24.812698 11023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:48907
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
I20260926 15:00:24.830390 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:40645
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:24.914878 11219 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:24.915124 11219 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:24.915164 11219 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:24.916370 11219 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:24.916466 11219 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:24.916517 11219 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:00:24.918027 11219 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:40645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11219
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:24.918308 11219 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:24.918545 11219 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:24.924254 11224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:24.924345 11219 server_base.cc:1034] running on GCE node
W20260926 15:00:24.924261 11227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:24.924261 11225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:24.924687 11219 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:24.924960 11219 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:24.926141 11219 hybrid_clock.cc:648] HybridClock initialized: now 1790434824926114 us; error 55 us; skew 500 ppm
Sep 26 15:00:24 dist-test-slave-jnhs krb5kdc[10978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434824, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:24.928809 11219 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:00:24.929346 11219 webserver.cc:460] Webserver started at http://127.8.102.2:39149/ using document root <none> and password file <none>
I20260926 15:00:24.929597 11219 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:24.929668 11219 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:24.929816 11219 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:24.930974 11219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "ff7ce911af764ade92afc7f27fc9c65b"
format_stamp: "Formatted at 2026-09-26 15:00:24 on dist-test-slave-jnhs"
server_key: "587bc5541f32b71615c0c1f474bf141a"
server_key_iv: "d9162a1e6f5a5fcbd1b57434bcae0f7f"
server_key_version: "encryptionkey@0"
I20260926 15:00:24.931344 11219 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "ff7ce911af764ade92afc7f27fc9c65b"
format_stamp: "Formatted at 2026-09-26 15:00:24 on dist-test-slave-jnhs"
server_key: "587bc5541f32b71615c0c1f474bf141a"
server_key_iv: "d9162a1e6f5a5fcbd1b57434bcae0f7f"
server_key_version: "encryptionkey@0"
I20260926 15:00:24.932686 11219 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:00:24.933422 11234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:24.933585 11219 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 15:00:24.933688 11219 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "ff7ce911af764ade92afc7f27fc9c65b"
format_stamp: "Formatted at 2026-09-26 15:00:24 on dist-test-slave-jnhs"
server_key: "587bc5541f32b71615c0c1f474bf141a"
server_key_iv: "d9162a1e6f5a5fcbd1b57434bcae0f7f"
server_key_version: "encryptionkey@0"
I20260926 15:00:24.933766 11219 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:24.950987 11219 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:24.953444 11219 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:24.953625 11219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:24.953867 11219 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:24.954197 11219 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:24.954249 11219 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:24.954285 11219 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:24.954313 11219 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:24.960707 11219 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:34365
I20260926 15:00:24.960752 11347 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:34365 every 8 connection(s)
I20260926 15:00:24.961084 11219 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:00:24.966005 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 11219
I20260926 15:00:24.966100 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:00:24.966324 8600 external_mini_cluster.cc:1428] Setting key 7251ef7e35189d3c3feaebde5e953e30
Sep 26 15:00:24 dist-test-slave-jnhs krb5kdc[10978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434824, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:24.971498 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:24.961817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:54659 (local address 127.8.102.62:40645)
0926 15:00:24.961867 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:24.961869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:24.962066 (+ 197us) server_negotiation.cc:407] Connection header received
0926 15:00:24.962565 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:24.962566 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:24.962576 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:24.962694 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:24.963658 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:24.964143 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:24.965134 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:24.965321 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:24.965379 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:24.967854 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:24.967856 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:24.967857 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:24.967866 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:24.969787 (+ 1921us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:24.970288 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:24.970290 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:24.970290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:24.970358 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:24.970763 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:24.970764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:24.970764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:24.970937 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:24.971013 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:24.971311 (+ 298us) server_negotiation.cc:299] Negotiation successful
0926 15:00:24.971404 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 15:00:24.971740 11350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:24.961773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:40645 (local address 127.8.102.2:54659)
0926 15:00:24.961932 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:24.961974 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:24.962454 (+ 480us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:24.962753 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:24.962756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:24.963350 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:24.963561 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:24.963567 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:24.964232 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:24.964233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:24.965048 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:24.965050 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:24.965396 (+ 346us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:24.965397 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:24.965490 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:24.966086 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:24.966100 (+ 14us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:24.967754 (+ 1654us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:24.969882 (+ 2128us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:24.969889 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:24.969892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:24.970203 (+ 311us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:24.970462 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:24.970463 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:24.970463 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:24.970577 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:24.971016 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:24.971021 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:24.971158 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 15:00:24.971270 (+ 112us) client_negotiation.cc:240] Negotiation successful
0926 15:00:24.971444 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":90,"threads_started":1}
I20260926 15:00:24.972254 11348 heartbeater.cc:347] Connected to a master server at 127.8.102.62:40645
I20260926 15:00:24.972355 11348 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:24.972556 11348 heartbeater.cc:511] Master 127.8.102.62:40645 requested a full tablet report, sending...
I20260926 15:00:24.972950 11023 ts_manager.cc:194] Registered new tserver with Master: ff7ce911af764ade92afc7f27fc9c65b (127.8.102.2:34365)
I20260926 15:00:24.973343 11023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:54659
WARNING: no policy specified for kudu/127.8.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
I20260926 15:00:24.999334 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:40645
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:45709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:25.082185 11353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:25.082438 11353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:25.082479 11353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:25.083817 11353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:25.083891 11353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:25.083937 11353 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:00:25.085435 11353 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:45709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:40645
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:25.085721 11353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:25.085953 11353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:25.091604 11359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:25.091588 11361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:25.091674 11353 server_base.cc:1034] running on GCE node
W20260926 15:00:25.091784 11358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:25.092007 11353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:25.092267 11353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:25.093485 11353 hybrid_clock.cc:648] HybridClock initialized: now 1790434825093459 us; error 35 us; skew 500 ppm
Sep 26 15:00:25 dist-test-slave-jnhs krb5kdc[10978](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434825, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:25.096271 11353 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:00:25.096802 11353 webserver.cc:460] Webserver started at http://127.8.102.3:45167/ using document root <none> and password file <none>
I20260926 15:00:25.097034 11353 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:25.097105 11353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:25.097254 11353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:25.098328 11353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "77090a5a9eaf40a788ff7a7dce356e12"
format_stamp: "Formatted at 2026-09-26 15:00:25 on dist-test-slave-jnhs"
server_key: "5d0704f98490e42cc437e621d18876b5"
server_key_iv: "eeddd8331f39cea700ea74488a0f20b7"
server_key_version: "encryptionkey@0"
I20260926 15:00:25.098764 11353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "77090a5a9eaf40a788ff7a7dce356e12"
format_stamp: "Formatted at 2026-09-26 15:00:25 on dist-test-slave-jnhs"
server_key: "5d0704f98490e42cc437e621d18876b5"
server_key_iv: "eeddd8331f39cea700ea74488a0f20b7"
server_key_version: "encryptionkey@0"
I20260926 15:00:25.100142 11353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:25.101027 11368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:25.101279 11353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:25.101388 11353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "77090a5a9eaf40a788ff7a7dce356e12"
format_stamp: "Formatted at 2026-09-26 15:00:25 on dist-test-slave-jnhs"
server_key: "5d0704f98490e42cc437e621d18876b5"
server_key_iv: "eeddd8331f39cea700ea74488a0f20b7"
server_key_version: "encryptionkey@0"
I20260926 15:00:25.101480 11353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:25.121632 11353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:25.123636 11353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:25.123801 11353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:25.124049 11353 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:25.124424 11353 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:25.124477 11353 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:25.124521 11353 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:25.124557 11353 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:25.130612 11353 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:35069
I20260926 15:00:25.130673 11481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:35069 every 8 connection(s)
I20260926 15:00:25.131017 11353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:00:25.134701 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 11353
I20260926 15:00:25.134817 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:00:25.134999 8600 external_mini_cluster.cc:1428] Setting key 772d2ed3aebace06ee1dcc0bfba25c9f
Sep 26 15:00:25 dist-test-slave-jnhs krb5kdc[10978](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434825, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:25.141391 11062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:25.131880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:47549 (local address 127.8.102.62:40645)
0926 15:00:25.131962 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:25.131964 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:25.132197 (+ 233us) server_negotiation.cc:407] Connection header received
0926 15:00:25.132743 (+ 546us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:25.132744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:25.132757 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:25.132809 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:25.133830 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:25.134343 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:25.135734 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:25.136048 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:25.136125 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:25.138565 (+ 2440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:25.138567 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:25.138568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:25.138576 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:25.139886 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:25.140351 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:25.140352 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:25.140353 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:25.140414 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:25.140699 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:25.140700 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:25.140701 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:25.140833 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:25.140910 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:25.141202 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 15:00:25.141296 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260926 15:00:25.141602 11484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:25.131870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:40645 (local address 127.8.102.3:47549)
0926 15:00:25.132063 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:25.132104 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:25.132623 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:25.132889 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:25.132892 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:25.133514 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:25.133720 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:25.133725 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:25.134473 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:25.134474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:25.135657 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:25.135661 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:25.136137 (+ 476us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:25.136139 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:25.136277 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:25.136894 (+ 617us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:25.136903 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:25.138458 (+ 1555us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:25.140020 (+ 1562us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:25.140026 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:25.140029 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:25.140268 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:25.140493 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:25.140494 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:25.140494 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:25.140628 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:25.140902 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:25.140908 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:25.141066 (+ 158us) client_negotiation.cc:769] Sending connection context
0926 15:00:25.141182 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 15:00:25.141352 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":87,"threads_started":1}
I20260926 15:00:25.142135 11482 heartbeater.cc:347] Connected to a master server at 127.8.102.62:40645
I20260926 15:00:25.142226 11482 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:25.142375 11482 heartbeater.cc:511] Master 127.8.102.62:40645 requested a full tablet report, sending...
I20260926 15:00:25.142791 11023 ts_manager.cc:194] Registered new tserver with Master: 77090a5a9eaf40a788ff7a7dce356e12 (127.8.102.3:35069)
I20260926 15:00:25.143204 11023 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:47549
I20260926 15:00:25.146950 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 15:00:25.159062 11062 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:25.157939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47412 (local address 127.8.102.62:40645)
0926 15:00:25.158001 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:25.158003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:25.158042 (+ 39us) server_negotiation.cc:407] Connection header received
0926 15:00:25.158137 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:25.158138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:25.158148 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:25.158205 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:25.158959 (+ 754us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47412: BlockingRecv error: recv got EOF from 127.0.0.1:47412 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 15:00:25.159971 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 11085
I20260926 15:00:25.165239 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 11219
I20260926 15:00:25.169461 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 11353
I20260926 15:00:25.173790 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 10991
2026-09-26T15:00:25Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1962 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:00:25 dist-test-slave-jnhs krb5kdc[11494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:00:25 dist-test-slave-jnhs krb5kdc[11494](info): set up 2 sockets
Sep 26 15:00:25 dist-test-slave-jnhs krb5kdc[11494](info): commencing operation
krb5kdc: starting...
W20260926 15:00:26.231118 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.036s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:00:26 dist-test-slave-jnhs krb5kdc[11494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:00:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:00:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:00:26.367101 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:41975
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:40161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:41975
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:26.449719 11507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:26.449940 11507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:26.449965 11507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:26.451198 11507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:26.451246 11507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:26.451260 11507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:26.451273 11507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:26.451285 11507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:00:26.452936 11507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:40161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:41975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:41975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:26.453220 11507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:26.453488 11507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:26.459316 11512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:26.459297 11515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:26.459298 11513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:26.459538 11507 server_base.cc:1034] running on GCE node
I20260926 15:00:26.459879 11507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:26.460263 11507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:26.461468 11507 hybrid_clock.cc:648] HybridClock initialized: now 1790434826461445 us; error 33 us; skew 500 ppm
Sep 26 15:00:26 dist-test-slave-jnhs krb5kdc[11494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434826, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:26.472337 11507 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:00:26.473013 11507 webserver.cc:460] Webserver started at http://127.8.102.62:40349/ using document root <none> and password file <none>
I20260926 15:00:26.473300 11507 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:26.473381 11507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:26.473552 11507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:26.474879 11507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "c6026232f09a48d29265f309121f8160"
format_stamp: "Formatted at 2026-09-26 15:00:26 on dist-test-slave-jnhs"
server_key: "3821af01188a757b773b7e2bc4d33928"
server_key_iv: "eb8f315987cd907ef04217d42b2ec0ff"
server_key_version: "encryptionkey@0"
I20260926 15:00:26.475283 11507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "c6026232f09a48d29265f309121f8160"
format_stamp: "Formatted at 2026-09-26 15:00:26 on dist-test-slave-jnhs"
server_key: "3821af01188a757b773b7e2bc4d33928"
server_key_iv: "eb8f315987cd907ef04217d42b2ec0ff"
server_key_version: "encryptionkey@0"
I20260926 15:00:26.476788 11507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:00:26.477727 11522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:26.478065 11507 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:26.478188 11507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "c6026232f09a48d29265f309121f8160"
format_stamp: "Formatted at 2026-09-26 15:00:26 on dist-test-slave-jnhs"
server_key: "3821af01188a757b773b7e2bc4d33928"
server_key_iv: "eb8f315987cd907ef04217d42b2ec0ff"
server_key_version: "encryptionkey@0"
I20260926 15:00:26.478271 11507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:26.503302 11507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:26.505877 11507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:26.506060 11507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:26.510591 11507 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:41975
I20260926 15:00:26.510733 11574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:41975 every 8 connection(s)
I20260926 15:00:26.511243 11507 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:00:26.511981 11575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:26.512785 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 11507
I20260926 15:00:26.512863 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:00:26.513070 8600 external_mini_cluster.cc:1428] Setting key 120b852b32a05f515d115401eef91302
I20260926 15:00:26.515434 11575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160: Bootstrap starting.
I20260926 15:00:26.516161 11575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:26.516511 11575 log.cc:824] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:26.517309 11575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160: No bootstrap required, opened a new log
I20260926 15:00:26.519500 11575 raft_consensus.cc:348] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6026232f09a48d29265f309121f8160" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41975 } }
I20260926 15:00:26.519718 11575 raft_consensus.cc:374] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:26.519757 11575 raft_consensus.cc:729] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6026232f09a48d29265f309121f8160, State: Initialized, Role: FOLLOWER
I20260926 15:00:26.519913 11575 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6026232f09a48d29265f309121f8160" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41975 } }
I20260926 15:00:26.520004 11575 raft_consensus.cc:388] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:00:26.520040 11575 raft_consensus.cc:482] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:00:26.520092 11575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 15:00:26 dist-test-slave-jnhs krb5kdc[11494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:26.520743 11575 raft_consensus.cc:504] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6026232f09a48d29265f309121f8160" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41975 } }
I20260926 15:00:26.520838 11575 leader_election.cc:302] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6026232f09a48d29265f309121f8160; no voters:
I20260926 15:00:26.521090 11575 leader_election.cc:288] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:00:26.521255 11580 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:26.521415 11575 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:26.521417 11580 raft_consensus.cc:686] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160 [term 1 LEADER]: Becoming Leader. State: Replica: c6026232f09a48d29265f309121f8160, State: Running, Role: LEADER
I20260926 15:00:26.521626 11580 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6026232f09a48d29265f309121f8160" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41975 } }
I20260926 15:00:26.522153 11581 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6026232f09a48d29265f309121f8160" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6026232f09a48d29265f309121f8160" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41975 } } }
I20260926 15:00:26.522365 11581 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:26.522228 11582 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6026232f09a48d29265f309121f8160. Latest consensus state: current_term: 1 leader_uuid: "c6026232f09a48d29265f309121f8160" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6026232f09a48d29265f309121f8160" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41975 } } }
I20260926 15:00:26.522596 11582 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160 [sys.catalog]: This master's current role is: LEADER
W20260926 15:00:26.523463 11595 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:00:26.523527 11595 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:00:26.523579 11590 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:00:26.523757 11590 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:00:26.525344 11578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:26.513672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38752 (local address 127.8.102.62:41975)
0926 15:00:26.513843 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:26.513852 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:26.513879 (+ 27us) server_negotiation.cc:407] Connection header received
0926 15:00:26.514287 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:26.514294 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:26.514342 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:26.514469 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:26.515461 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:26.516536 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:26.517375 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:26.517837 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:26.517963 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:26.520737 (+ 2774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:26.520742 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:26.520743 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:26.520757 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:26.523491 (+ 2734us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:26.523998 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:26.524001 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:26.524001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:26.524095 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:26.524339 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:26.524340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:26.524340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:26.524616 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:26.524779 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:26.524902 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 15:00:26.525066 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":82,"threads_started":1}
I20260926 15:00:26.525355 11590 catalog_manager.cc:1347] Generated new cluster ID: d9c97d142e864b9fb274c481871c11f8
I20260926 15:00:26.525465 11590 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:00:26.526386 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:00:26.544483 11578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:26.540048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38762 (local address 127.8.102.62:41975)
0926 15:00:26.540093 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:26.540095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:26.540194 (+ 99us) server_negotiation.cc:407] Connection header received
0926 15:00:26.540265 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:26.540266 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:26.540278 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:26.540360 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:26.540800 (+ 440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:26.541594 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:26.542748 (+ 1154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:26.543087 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:26.543161 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:26.543953 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:26.543955 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:26.543955 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:00:26.544204 (+ 249us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:26.544225 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:00:26.544232 (+ 7us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:26.544294 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:26.544361 (+ 67us) server_negotiation.cc:299] Negotiation successful
0926 15:00:26.544389 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 15:00:26.555063 11590 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:00:26.555717 11590 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:00:26.560948 11590 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c6026232f09a48d29265f309121f8160: Generated new TSK 0
I20260926 15:00:26.561173 11590 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:00:26.568918 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 11507
2026-09-26T15:00:26Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1392 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:00:26 dist-test-slave-jnhs krb5kdc[11608](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:00:26 dist-test-slave-jnhs krb5kdc[11608](info): set up 2 sockets
Sep 26 15:00:26 dist-test-slave-jnhs krb5kdc[11608](info): commencing operation
krb5kdc: starting...
W20260926 15:00:27.630584 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.043s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:00:27 dist-test-slave-jnhs krb5kdc[11608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:00:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:00:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:00:27.758980 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:44477
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:44477
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:27.841277 11621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:27.841550 11621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:27.841590 11621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:27.842837 11621 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:27.842918 11621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:27.842942 11621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:27.843001 11621 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:27.843044 11621 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:00:27.844856 11621 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:44477
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:44477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:27.845346 11621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:27.845695 11621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:27.851120 11626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:27.851128 11629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:27.851128 11627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:27.851362 11621 server_base.cc:1034] running on GCE node
I20260926 15:00:27.851681 11621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:27.851992 11621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:27.853166 11621 hybrid_clock.cc:648] HybridClock initialized: now 1790434827853143 us; error 39 us; skew 500 ppm
Sep 26 15:00:27 dist-test-slave-jnhs krb5kdc[11608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434827, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:27.855841 11621 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:00:27.856308 11621 webserver.cc:460] Webserver started at http://127.8.102.62:40795/ using document root <none> and password file <none>
I20260926 15:00:27.856560 11621 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:27.856630 11621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:27.856778 11621 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:27.857867 11621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "86512d49a3af4555b2b4b98e1011ecc7"
format_stamp: "Formatted at 2026-09-26 15:00:27 on dist-test-slave-jnhs"
server_key: "3b080f2df7268f6e141cdb2ecda1e359"
server_key_iv: "e7a3e67b5f163000b123979375e1e58f"
server_key_version: "encryptionkey@0"
I20260926 15:00:27.858246 11621 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "86512d49a3af4555b2b4b98e1011ecc7"
format_stamp: "Formatted at 2026-09-26 15:00:27 on dist-test-slave-jnhs"
server_key: "3b080f2df7268f6e141cdb2ecda1e359"
server_key_iv: "e7a3e67b5f163000b123979375e1e58f"
server_key_version: "encryptionkey@0"
I20260926 15:00:27.859783 11621 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 15:00:27.860600 11636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:27.860816 11621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 15:00:27.860924 11621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "86512d49a3af4555b2b4b98e1011ecc7"
format_stamp: "Formatted at 2026-09-26 15:00:27 on dist-test-slave-jnhs"
server_key: "3b080f2df7268f6e141cdb2ecda1e359"
server_key_iv: "e7a3e67b5f163000b123979375e1e58f"
server_key_version: "encryptionkey@0"
I20260926 15:00:27.861001 11621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:27.907404 11621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:27.909629 11621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:27.909811 11621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:27.913887 11621 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:44477
I20260926 15:00:27.913937 11688 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:44477 every 8 connection(s)
I20260926 15:00:27.914368 11621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:00:27.915457 11689 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:27.917838 11689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7: Bootstrap starting.
I20260926 15:00:27.918464 11689 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:27.918794 11689 log.cc:824] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:27.919528 11689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7: No bootstrap required, opened a new log
I20260926 15:00:27.921144 11689 raft_consensus.cc:348] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86512d49a3af4555b2b4b98e1011ecc7" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44477 } }
I20260926 15:00:27.921288 11689 raft_consensus.cc:374] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:27.921310 11689 raft_consensus.cc:729] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86512d49a3af4555b2b4b98e1011ecc7, State: Initialized, Role: FOLLOWER
I20260926 15:00:27.921494 11689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86512d49a3af4555b2b4b98e1011ecc7" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44477 } }
I20260926 15:00:27.921557 11689 raft_consensus.cc:388] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:00:27.921603 11689 raft_consensus.cc:482] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:00:27.921655 11689 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:27.922503 11689 raft_consensus.cc:504] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86512d49a3af4555b2b4b98e1011ecc7" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44477 } }
I20260926 15:00:27.922595 11689 leader_election.cc:302] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 86512d49a3af4555b2b4b98e1011ecc7; no voters:
I20260926 15:00:27.922827 11689 leader_election.cc:288] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:00:27.922865 11692 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:27.923072 11692 raft_consensus.cc:686] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7 [term 1 LEADER]: Becoming Leader. State: Replica: 86512d49a3af4555b2b4b98e1011ecc7, State: Running, Role: LEADER
I20260926 15:00:27.923130 11689 sys_catalog.cc:564] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:27.923179 11692 consensus_queue.cc:237] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86512d49a3af4555b2b4b98e1011ecc7" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44477 } }
I20260926 15:00:27.923558 11693 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "86512d49a3af4555b2b4b98e1011ecc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86512d49a3af4555b2b4b98e1011ecc7" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44477 } } }
I20260926 15:00:27.923558 11694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 86512d49a3af4555b2b4b98e1011ecc7. Latest consensus state: current_term: 1 leader_uuid: "86512d49a3af4555b2b4b98e1011ecc7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "86512d49a3af4555b2b4b98e1011ecc7" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44477 } } }
I20260926 15:00:27.923686 11693 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:27.923689 11694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:27.923946 11701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:00:27.924501 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 11621
I20260926 15:00:27.924576 11701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:00:27.924631 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:00:27.924842 8600 external_mini_cluster.cc:1428] Setting key 11222507dd0ca5443e36f104e78bc973
I20260926 15:00:27.926288 11701 catalog_manager.cc:1347] Generated new cluster ID: 87b3af95ed1f456fa0655c40ba11ead8
I20260926 15:00:27.926339 11701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 15:00:27 dist-test-slave-jnhs krb5kdc[11608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434827, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:27.936540 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:27.925330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60128 (local address 127.8.102.62:44477)
0926 15:00:27.925471 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:27.925481 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:27.925523 (+ 42us) server_negotiation.cc:407] Connection header received
0926 15:00:27.925625 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:27.925631 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:27.925676 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:27.925831 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:27.926620 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:27.927977 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:27.929084 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:27.929706 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:27.929803 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:27.932743 (+ 2940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:27.932748 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:27.932748 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:27.932762 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:27.934785 (+ 2023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:27.935204 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:27.935208 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:27.935208 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:27.935328 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:27.935541 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:27.935541 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:27.935542 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:27.935860 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:27.936010 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:27.936141 (+ 131us) server_negotiation.cc:299] Negotiation successful
0926 15:00:27.936279 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":66,"threads_started":1}
I20260926 15:00:27.937798 11701 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:00:27.938292 11701 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:00:27.945746 11701 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 86512d49a3af4555b2b4b98e1011ecc7: Generated new TSK 0
I20260926 15:00:27.945966 11701 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
I20260926 15:00:27.980150 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44477
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:28.065932 11715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:28.066195 11715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:28.066239 11715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:28.067515 11715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:28.067582 11715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:28.067631 11715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:00:28.069115 11715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:28.069386 11715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:28.069626 11715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 15:00:28.074885 11715 server_base.cc:1034] running on GCE node
W20260926 15:00:28.074885 11720 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:28.074954 11723 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:28.074919 11721 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:28.075287 11715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:28.075575 11715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:28.076749 11715 hybrid_clock.cc:648] HybridClock initialized: now 1790434828076739 us; error 32 us; skew 500 ppm
Sep 26 15:00:28 dist-test-slave-jnhs krb5kdc[11608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434828, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:28.079411 11715 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:00:28.079875 11715 webserver.cc:460] Webserver started at http://127.8.102.1:35627/ using document root <none> and password file <none>
I20260926 15:00:28.080127 11715 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:28.080206 11715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:28.080372 11715 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:28.081470 11715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "249a01bce6fb496a96232c8c7e1e19d1"
format_stamp: "Formatted at 2026-09-26 15:00:28 on dist-test-slave-jnhs"
server_key: "153ffa53e5683a5ba7ebef5bb48e2a57"
server_key_iv: "65c397aeb1748183415f898555aceefb"
server_key_version: "encryptionkey@0"
I20260926 15:00:28.081842 11715 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "249a01bce6fb496a96232c8c7e1e19d1"
format_stamp: "Formatted at 2026-09-26 15:00:28 on dist-test-slave-jnhs"
server_key: "153ffa53e5683a5ba7ebef5bb48e2a57"
server_key_iv: "65c397aeb1748183415f898555aceefb"
server_key_version: "encryptionkey@0"
I20260926 15:00:28.083364 11715 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 15:00:28.084175 11730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:28.084342 11715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:28.084448 11715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "249a01bce6fb496a96232c8c7e1e19d1"
format_stamp: "Formatted at 2026-09-26 15:00:28 on dist-test-slave-jnhs"
server_key: "153ffa53e5683a5ba7ebef5bb48e2a57"
server_key_iv: "65c397aeb1748183415f898555aceefb"
server_key_version: "encryptionkey@0"
I20260926 15:00:28.084523 11715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:28.096745 11715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:28.099164 11715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:28.099346 11715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:28.099596 11715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:28.099908 11715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:28.099964 11715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:28.100005 11715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:28.100056 11715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:28.105875 11715 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:33209
I20260926 15:00:28.105913 11843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:33209 every 8 connection(s)
I20260926 15:00:28.106254 11715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
Sep 26 15:00:28 dist-test-slave-jnhs krb5kdc[11608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434828, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:28.115760 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:28.107044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:38661 (local address 127.8.102.62:44477)
0926 15:00:28.107128 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:28.107131 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:28.107337 (+ 206us) server_negotiation.cc:407] Connection header received
0926 15:00:28.107885 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:28.107886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:28.107898 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:28.107961 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:28.108907 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:28.109408 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:28.110448 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:28.110712 (+ 264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:28.110774 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:28.112981 (+ 2207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:28.112983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:28.112983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:28.112991 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:28.114249 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:28.114733 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:28.114734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:28.114735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:28.114798 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:28.115068 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:28.115069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:28.115070 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:28.115202 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:28.115277 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:28.115562 (+ 285us) server_negotiation.cc:299] Negotiation successful
0926 15:00:28.115665 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 15:00:28.115847 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 11715
I20260926 15:00:28.115958 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:00:28.115958 11846 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:28.106982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44477 (local address 127.8.102.1:38661)
0926 15:00:28.107197 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:28.107242 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:28.107795 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:28.108046 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:28.108050 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:28.108629 (+ 579us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:28.108823 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:28.108828 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:28.109498 (+ 670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:28.109499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:28.110355 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:28.110357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:28.110793 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:28.110794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:28.110904 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:28.111465 (+ 561us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:28.111476 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:28.112877 (+ 1401us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:28.114365 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:28.114372 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:28.114374 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:28.114621 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:28.114880 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:28.114880 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:28.114881 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:28.114990 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:28.115278 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:28.115285 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:28.115424 (+ 139us) client_negotiation.cc:769] Sending connection context
0926 15:00:28.115526 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 15:00:28.115712 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":93,"threads_started":1}
I20260926 15:00:28.116169 8600 external_mini_cluster.cc:1428] Setting key 3f15d079cf4210718dc1c5719ea4007d
I20260926 15:00:28.116616 11844 heartbeater.cc:347] Connected to a master server at 127.8.102.62:44477
I20260926 15:00:28.116715 11844 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:28.116909 11844 heartbeater.cc:511] Master 127.8.102.62:44477 requested a full tablet report, sending...
I20260926 15:00:28.117499 11653 ts_manager.cc:194] Registered new tserver with Master: 249a01bce6fb496a96232c8c7e1e19d1 (127.8.102.1:33209)
I20260926 15:00:28.118459 11653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:38661
WARNING: no policy specified for kudu/127.8.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
I20260926 15:00:28.147150 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44477
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:28.230347 11849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:28.230616 11849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:28.230659 11849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:28.231931 11849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:28.232012 11849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:28.232064 11849 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:00:28.233626 11849 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11849
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:28.233933 11849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:28.234189 11849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:28.240020 11854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:28.240319 11857 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:28.240429 11855 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:28.240465 11849 server_base.cc:1034] running on GCE node
I20260926 15:00:28.240731 11849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:28.241026 11849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:28.242190 11849 hybrid_clock.cc:648] HybridClock initialized: now 1790434828242181 us; error 31 us; skew 500 ppm
Sep 26 15:00:28 dist-test-slave-jnhs krb5kdc[11608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434828, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:28.245223 11849 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:00:28.245756 11849 webserver.cc:460] Webserver started at http://127.8.102.2:35475/ using document root <none> and password file <none>
I20260926 15:00:28.246016 11849 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:28.246069 11849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:28.246253 11849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:28.247459 11849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "4ce9d3982c90428bafa772ff0675a7c1"
format_stamp: "Formatted at 2026-09-26 15:00:28 on dist-test-slave-jnhs"
server_key: "a8425173fee091256b48b4dc8a5dcbd9"
server_key_iv: "abd6d77058def70dac437f04e1c6bc13"
server_key_version: "encryptionkey@0"
I20260926 15:00:28.247840 11849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "4ce9d3982c90428bafa772ff0675a7c1"
format_stamp: "Formatted at 2026-09-26 15:00:28 on dist-test-slave-jnhs"
server_key: "a8425173fee091256b48b4dc8a5dcbd9"
server_key_iv: "abd6d77058def70dac437f04e1c6bc13"
server_key_version: "encryptionkey@0"
I20260926 15:00:28.249310 11849 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:00:28.250213 11864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:28.250399 11849 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:28.250507 11849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "4ce9d3982c90428bafa772ff0675a7c1"
format_stamp: "Formatted at 2026-09-26 15:00:28 on dist-test-slave-jnhs"
server_key: "a8425173fee091256b48b4dc8a5dcbd9"
server_key_iv: "abd6d77058def70dac437f04e1c6bc13"
server_key_version: "encryptionkey@0"
I20260926 15:00:28.250608 11849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:28.267068 11849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:28.269472 11849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:28.269647 11849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:28.269891 11849 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:28.270236 11849 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:28.270293 11849 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:28.270336 11849 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:28.270363 11849 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:28.276407 11849 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:41139
I20260926 15:00:28.276455 11977 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:41139 every 8 connection(s)
I20260926 15:00:28.276824 11849 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:00:28.282305 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 11849
I20260926 15:00:28.282418 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:00:28.282663 8600 external_mini_cluster.cc:1428] Setting key 82687b59d4cabb0f41629ef6a077e1f3
Sep 26 15:00:28 dist-test-slave-jnhs krb5kdc[11608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434828, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:28.288365 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:28.277572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:59193 (local address 127.8.102.62:44477)
0926 15:00:28.277638 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:28.277640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:28.277867 (+ 227us) server_negotiation.cc:407] Connection header received
0926 15:00:28.278355 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:28.278356 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:28.278364 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:28.278430 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:28.279368 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:28.279877 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:28.281106 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:28.281480 (+ 374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:28.281555 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:28.284689 (+ 3134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:28.284690 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:28.284691 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:28.284699 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:28.286739 (+ 2040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:28.287228 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:28.287230 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:28.287231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:28.287335 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:28.287611 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:28.287612 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:28.287613 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:28.287753 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:28.287843 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:28.288164 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 15:00:28.288261 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 15:00:28.288628 11980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:28.277564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44477 (local address 127.8.102.2:59193)
0926 15:00:28.277732 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:28.277777 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:28.278272 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:28.278485 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:28.278488 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:28.279060 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:28.279255 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:28.279259 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:28.279989 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:28.279990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:28.281025 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:28.281028 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:28.281627 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:28.281628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:28.281747 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:28.282489 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:28.282500 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:28.284542 (+ 2042us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:28.286842 (+ 2300us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:28.286852 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:28.286854 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:28.287136 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:28.287396 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:28.287397 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:28.287397 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:28.287530 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:28.287832 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:28.287839 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:28.288004 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 15:00:28.288129 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 15:00:28.288319 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":99,"threads_started":1}
I20260926 15:00:28.289336 11978 heartbeater.cc:347] Connected to a master server at 127.8.102.62:44477
I20260926 15:00:28.289458 11978 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:28.289672 11978 heartbeater.cc:511] Master 127.8.102.62:44477 requested a full tablet report, sending...
I20260926 15:00:28.290156 11653 ts_manager.cc:194] Registered new tserver with Master: 4ce9d3982c90428bafa772ff0675a7c1 (127.8.102.2:41139)
I20260926 15:00:28.290707 11653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:59193
WARNING: no policy specified for kudu/127.8.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
I20260926 15:00:28.311128 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44477
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46673
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:28.393833 11983 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:28.394047 11983 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:28.394079 11983 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:28.395224 11983 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:28.395262 11983 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:28.395289 11983 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:00:28.396662 11983 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46673
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44477
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.11983
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:28.396854 11983 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:28.397037 11983 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:28.402191 11989 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:28.402287 11983 server_base.cc:1034] running on GCE node
W20260926 15:00:28.402220 11988 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:28.402225 11991 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:28.402567 11983 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:28.402841 11983 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:28.404008 11983 hybrid_clock.cc:648] HybridClock initialized: now 1790434828403994 us; error 47 us; skew 500 ppm
Sep 26 15:00:28 dist-test-slave-jnhs krb5kdc[11608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434828, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:28.406590 11983 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:00:28.407104 11983 webserver.cc:460] Webserver started at http://127.8.102.3:35533/ using document root <none> and password file <none>
I20260926 15:00:28.407332 11983 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:28.407398 11983 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:28.407536 11983 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:28.408595 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "ce52a9db720b4c7a8ac2a3a2f105d560"
format_stamp: "Formatted at 2026-09-26 15:00:28 on dist-test-slave-jnhs"
server_key: "191460be7b9494f93d6de1ae517c829c"
server_key_iv: "0b6c707c73858d99d3d8790b0d24ad7b"
server_key_version: "encryptionkey@0"
I20260926 15:00:28.408950 11983 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "ce52a9db720b4c7a8ac2a3a2f105d560"
format_stamp: "Formatted at 2026-09-26 15:00:28 on dist-test-slave-jnhs"
server_key: "191460be7b9494f93d6de1ae517c829c"
server_key_iv: "0b6c707c73858d99d3d8790b0d24ad7b"
server_key_version: "encryptionkey@0"
I20260926 15:00:28.410283 11983 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 15:00:28.411100 11998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:28.411273 11983 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:28.411370 11983 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "ce52a9db720b4c7a8ac2a3a2f105d560"
format_stamp: "Formatted at 2026-09-26 15:00:28 on dist-test-slave-jnhs"
server_key: "191460be7b9494f93d6de1ae517c829c"
server_key_iv: "0b6c707c73858d99d3d8790b0d24ad7b"
server_key_version: "encryptionkey@0"
I20260926 15:00:28.411449 11983 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:28.437237 11983 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:28.439536 11983 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:28.439704 11983 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:28.439944 11983 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:28.440269 11983 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:28.440322 11983 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:28.440367 11983 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:28.440405 11983 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:28.446547 11983 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:46559
I20260926 15:00:28.446595 12111 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:46559 every 8 connection(s)
I20260926 15:00:28.446936 11983 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
Sep 26 15:00:28 dist-test-slave-jnhs krb5kdc[11608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434828, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:28.456686 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 11983
I20260926 15:00:28.456776 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:00:28.456992 8600 external_mini_cluster.cc:1428] Setting key 333e4a9451bebed31747cb847b56a8b6
I20260926 15:00:28.457612 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:28.447751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:33741 (local address 127.8.102.62:44477)
0926 15:00:28.447818 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:28.447820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:28.448053 (+ 233us) server_negotiation.cc:407] Connection header received
0926 15:00:28.448692 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:28.448693 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:28.448707 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:28.448769 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:28.449821 (+ 1052us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:28.450377 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:28.451570 (+ 1193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:28.451797 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:28.451901 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:28.454473 (+ 2572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:28.454475 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:28.454475 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:28.454482 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:28.455854 (+ 1372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:28.456262 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:28.456263 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:28.456263 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:28.456326 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:28.456688 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:28.456690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:28.456690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:28.456868 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:28.456959 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:28.457398 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 15:00:28.457511 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 15:00:28.457805 12114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:28.447765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44477 (local address 127.8.102.3:33741)
0926 15:00:28.447918 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:28.447958 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:28.448571 (+ 613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:28.448855 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:28.448858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:28.449492 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:28.449701 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:28.449706 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:28.450489 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:28.450490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:28.451459 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:28.451462 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:28.451904 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:28.451905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:28.452059 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:28.452680 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:28.452690 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:28.454335 (+ 1645us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:28.455942 (+ 1607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:28.455948 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:28.455950 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:28.456183 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:28.456397 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:28.456398 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:28.456398 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:28.456566 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:28.456963 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:28.456969 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:28.457192 (+ 223us) client_negotiation.cc:769] Sending connection context
0926 15:00:28.457325 (+ 133us) client_negotiation.cc:240] Negotiation successful
0926 15:00:28.457547 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
I20260926 15:00:28.458235 12112 heartbeater.cc:347] Connected to a master server at 127.8.102.62:44477
I20260926 15:00:28.458353 12112 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:28.458598 12112 heartbeater.cc:511] Master 127.8.102.62:44477 requested a full tablet report, sending...
I20260926 15:00:28.459015 11653 ts_manager.cc:194] Registered new tserver with Master: ce52a9db720b4c7a8ac2a3a2f105d560 (127.8.102.3:46559)
I20260926 15:00:28.459409 11653 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:33741
I20260926 15:00:28.469166 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 15:00:28 dist-test-slave-jnhs krb5kdc[11608](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434828, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 26 15:00:28 dist-test-slave-jnhs krb5kdc[11608](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434828, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:28.492951 11711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:28.485577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60134 (local address 127.8.102.62:44477)
0926 15:00:28.485669 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:28.485672 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:28.485797 (+ 125us) server_negotiation.cc:407] Connection header received
0926 15:00:28.485846 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:28.485847 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:28.485857 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:28.485916 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:28.486462 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:28.487084 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:28.487867 (+ 783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:28.488087 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:28.488150 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:28.490500 (+ 2350us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:28.490501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:28.490502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:28.490509 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:28.491874 (+ 1365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:28.492191 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:28.492192 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:28.492192 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:28.492248 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:28.492422 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:28.492423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:28.492423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:28.492590 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:28.492674 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:28.492765 (+ 91us) server_negotiation.cc:299] Negotiation successful
0926 15:00:28.492830 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
W20260926 15:00:28.493247 11653 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:60134
I20260926 15:00:28.493731 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 11715
I20260926 15:00:28.498298 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 11849
I20260926 15:00:28.502720 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 11983
I20260926 15:00:28.507048 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 11621
2026-09-26T15:00:28Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1939 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:00:28 dist-test-slave-jnhs krb5kdc[12124](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:00:28 dist-test-slave-jnhs krb5kdc[12124](info): set up 2 sockets
Sep 26 15:00:28 dist-test-slave-jnhs krb5kdc[12124](info): commencing operation
krb5kdc: starting...
W20260926 15:00:29.550601 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:00:29 dist-test-slave-jnhs krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:00:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:00:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:00:29.677977 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:39647
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:39647
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:29.759502 12137 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:29.759728 12137 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:29.759752 12137 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:29.760941 12137 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:29.760990 12137 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:29.761005 12137 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:29.761018 12137 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:29.761032 12137 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:00:29.762573 12137 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:39647
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:39647
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12137
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:29.762871 12137 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:29.763111 12137 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:29.767974 12143 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:29.767976 12142 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:29.767975 12145 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:29.768157 12137 server_base.cc:1034] running on GCE node
I20260926 15:00:29.768486 12137 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:29.768769 12137 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:29.769905 12137 hybrid_clock.cc:648] HybridClock initialized: now 1790434829769902 us; error 42 us; skew 500 ppm
Sep 26 15:00:29 dist-test-slave-jnhs krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434829, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:29.772352 12137 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:00:29.772876 12137 webserver.cc:460] Webserver started at http://127.8.102.62:40463/ using document root <none> and password file <none>
I20260926 15:00:29.773115 12137 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:29.773161 12137 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:29.773329 12137 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:29.774389 12137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "3b3240b0f9ff4ce9a6c9d57b442cfd30"
format_stamp: "Formatted at 2026-09-26 15:00:29 on dist-test-slave-jnhs"
server_key: "47189fbd7fc8554f731038d19a891a27"
server_key_iv: "90fe2024e4969cfd5596b8d2ad3f5785"
server_key_version: "encryptionkey@0"
I20260926 15:00:29.774778 12137 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "3b3240b0f9ff4ce9a6c9d57b442cfd30"
format_stamp: "Formatted at 2026-09-26 15:00:29 on dist-test-slave-jnhs"
server_key: "47189fbd7fc8554f731038d19a891a27"
server_key_iv: "90fe2024e4969cfd5596b8d2ad3f5785"
server_key_version: "encryptionkey@0"
I20260926 15:00:29.776134 12137 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:29.776947 12152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:29.777159 12137 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:29.777249 12137 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "3b3240b0f9ff4ce9a6c9d57b442cfd30"
format_stamp: "Formatted at 2026-09-26 15:00:29 on dist-test-slave-jnhs"
server_key: "47189fbd7fc8554f731038d19a891a27"
server_key_iv: "90fe2024e4969cfd5596b8d2ad3f5785"
server_key_version: "encryptionkey@0"
I20260926 15:00:29.777323 12137 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:29.783878 12137 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:29.785930 12137 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:29.786079 12137 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:29.790272 12137 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:39647
I20260926 15:00:29.790306 12204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:39647 every 8 connection(s)
I20260926 15:00:29.790843 12137 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:00:29.791505 12205 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:29.792887 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 12137
I20260926 15:00:29.793041 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:00:29.793239 8600 external_mini_cluster.cc:1428] Setting key 6d32b59755e27f65593a12fbb0a3300d
I20260926 15:00:29.794078 12205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30: Bootstrap starting.
I20260926 15:00:29.794739 12205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:29.795082 12205 log.cc:824] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:29.795809 12205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30: No bootstrap required, opened a new log
I20260926 15:00:29.798048 12205 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3240b0f9ff4ce9a6c9d57b442cfd30" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 39647 } }
I20260926 15:00:29.798251 12205 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:29.798292 12205 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b3240b0f9ff4ce9a6c9d57b442cfd30, State: Initialized, Role: FOLLOWER
I20260926 15:00:29.798475 12205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3240b0f9ff4ce9a6c9d57b442cfd30" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 39647 } }
I20260926 15:00:29.798590 12205 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:00:29.798659 12205 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:00:29.798715 12205 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:29.799516 12205 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3240b0f9ff4ce9a6c9d57b442cfd30" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 39647 } }
I20260926 15:00:29.799664 12205 leader_election.cc:302] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b3240b0f9ff4ce9a6c9d57b442cfd30; no voters:
I20260926 15:00:29.799839 12205 leader_election.cc:288] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:00:29.799934 12210 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:29.800122 12205 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:29.800169 12210 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30 [term 1 LEADER]: Becoming Leader. State: Replica: 3b3240b0f9ff4ce9a6c9d57b442cfd30, State: Running, Role: LEADER
I20260926 15:00:29.800352 12210 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3240b0f9ff4ce9a6c9d57b442cfd30" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 39647 } }
Sep 26 15:00:29 dist-test-slave-jnhs krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434829, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:29.800731 12211 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b3240b0f9ff4ce9a6c9d57b442cfd30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3240b0f9ff4ce9a6c9d57b442cfd30" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 39647 } } }
I20260926 15:00:29.800809 12211 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:29.800979 12212 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b3240b0f9ff4ce9a6c9d57b442cfd30. Latest consensus state: current_term: 1 leader_uuid: "3b3240b0f9ff4ce9a6c9d57b442cfd30" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b3240b0f9ff4ce9a6c9d57b442cfd30" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 39647 } } }
I20260926 15:00:29.801036 12212 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:29.801458 12219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:00:29.802021 12219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:00:29.803720 12219 catalog_manager.cc:1347] Generated new cluster ID: 7e35b0c8d7e144a88b7babfe4b06b7b5
I20260926 15:00:29.803771 12219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:00:29.805421 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:29.793815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43962 (local address 127.8.102.62:39647)
0926 15:00:29.793989 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:29.794005 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:29.794032 (+ 27us) server_negotiation.cc:407] Connection header received
0926 15:00:29.794486 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:29.794493 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:29.794556 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:29.794752 (+ 196us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:29.795587 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:29.796551 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:29.797673 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:29.798104 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:29.798189 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:29.801304 (+ 3115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:29.801308 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:29.801308 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:29.801321 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:29.803345 (+ 2024us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:29.803753 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:29.803756 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:29.803756 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:29.803864 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:29.804116 (+ 252us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:29.804117 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:29.804117 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:29.804427 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:29.804572 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:29.805026 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 15:00:29.805205 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":99,"threads_started":1}
I20260926 15:00:29.825304 12219 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:00:29.826104 12219 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:00:29.829533 12219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b3240b0f9ff4ce9a6c9d57b442cfd30: Generated new TSK 0
I20260926 15:00:29.829763 12219 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
I20260926 15:00:29.867676 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:39647
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:29.951699 12231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:29.951946 12231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:29.951985 12231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:29.953162 12231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:29.953231 12231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:29.953272 12231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:00:29.954847 12231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:39647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:29.955134 12231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:29.955376 12231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:29.960551 12236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:29.960551 12237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:29.960603 12231 server_base.cc:1034] running on GCE node
W20260926 15:00:29.960563 12239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:29.960966 12231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:29.961257 12231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:29.962423 12231 hybrid_clock.cc:648] HybridClock initialized: now 1790434829962409 us; error 34 us; skew 500 ppm
Sep 26 15:00:29 dist-test-slave-jnhs krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434829, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:29.965134 12231 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:00:29.965664 12231 webserver.cc:460] Webserver started at http://127.8.102.1:36449/ using document root <none> and password file <none>
I20260926 15:00:29.966068 12231 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:29.966147 12231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:29.966298 12231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:29.967492 12231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "16d4af750eb64728b399909d0da36599"
format_stamp: "Formatted at 2026-09-26 15:00:29 on dist-test-slave-jnhs"
server_key: "d23f9cabdbf723deb39d70d437747e45"
server_key_iv: "cbb2109fe6eb73519307fbb37d581355"
server_key_version: "encryptionkey@0"
I20260926 15:00:29.967840 12231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "16d4af750eb64728b399909d0da36599"
format_stamp: "Formatted at 2026-09-26 15:00:29 on dist-test-slave-jnhs"
server_key: "d23f9cabdbf723deb39d70d437747e45"
server_key_iv: "cbb2109fe6eb73519307fbb37d581355"
server_key_version: "encryptionkey@0"
I20260926 15:00:29.969244 12231 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:00:29.970111 12246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:29.970278 12231 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:29.970366 12231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "16d4af750eb64728b399909d0da36599"
format_stamp: "Formatted at 2026-09-26 15:00:29 on dist-test-slave-jnhs"
server_key: "d23f9cabdbf723deb39d70d437747e45"
server_key_iv: "cbb2109fe6eb73519307fbb37d581355"
server_key_version: "encryptionkey@0"
I20260926 15:00:29.970432 12231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:29.977216 12231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:29.979465 12231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:29.979575 12231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:29.979739 12231 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:29.980075 12231 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:29.980113 12231 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:29.980134 12231 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:29.980182 12231 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:29.985836 12231 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:37521
I20260926 15:00:29.985914 12359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:37521 every 8 connection(s)
I20260926 15:00:29.986255 12231 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
Sep 26 15:00:29 dist-test-slave-jnhs krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434829, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:29.993291 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 12231
I20260926 15:00:29.993425 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:00:29.993626 8600 external_mini_cluster.cc:1428] Setting key f815b681f1dd09f499b75afe1d5e546f
I20260926 15:00:29.996012 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:29.987072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:42441 (local address 127.8.102.62:39647)
0926 15:00:29.987118 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:29.987121 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:29.987290 (+ 169us) server_negotiation.cc:407] Connection header received
0926 15:00:29.988225 (+ 935us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:29.988226 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:29.988236 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:29.988299 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:29.988894 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:29.989400 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:29.990352 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:29.990545 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:29.990600 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:29.992900 (+ 2300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:29.992901 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:29.992902 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:29.992908 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:29.994577 (+ 1669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:29.995016 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:29.995017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:29.995018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:29.995087 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:29.995347 (+ 260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:29.995348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:29.995349 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:29.995473 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:29.995549 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:29.995830 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 15:00:29.995922 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 15:00:29.996219 12362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:29.986980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:39647 (local address 127.8.102.1:42441)
0926 15:00:29.987153 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:29.987197 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:29.988127 (+ 930us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:29.988369 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:29.988372 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:29.988623 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:29.988817 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:29.988821 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:29.989478 (+ 657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:29.989478 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:29.990277 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:29.990279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:29.990617 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:29.990617 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:29.990746 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:29.991250 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:29.991258 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:29.992798 (+ 1540us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:29.994681 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:29.994689 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:29.994691 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:29.994934 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:29.995155 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:29.995156 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:29.995157 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:29.995268 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:29.995541 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:29.995547 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:29.995695 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 15:00:29.995799 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 15:00:29.995972 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":105,"threads_started":1}
I20260926 15:00:29.996757 12360 heartbeater.cc:347] Connected to a master server at 127.8.102.62:39647
I20260926 15:00:29.996860 12360 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:29.997048 12360 heartbeater.cc:511] Master 127.8.102.62:39647 requested a full tablet report, sending...
I20260926 15:00:29.997629 12169 ts_manager.cc:194] Registered new tserver with Master: 16d4af750eb64728b399909d0da36599 (127.8.102.1:37521)
I20260926 15:00:29.998376 12169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:42441
WARNING: no policy specified for kudu/127.8.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
I20260926 15:00:30.022611 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:39647
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:30.108996 12365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:30.109256 12365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:30.109293 12365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:30.110473 12365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:30.110553 12365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:30.110597 12365 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:00:30.112175 12365 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:39647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:30.112464 12365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:30.112709 12365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:30.117950 12370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:30.118080 12371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:30.117934 12373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:30.118158 12365 server_base.cc:1034] running on GCE node
I20260926 15:00:30.118544 12365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:30.118865 12365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:30.120028 12365 hybrid_clock.cc:648] HybridClock initialized: now 1790434830119993 us; error 64 us; skew 500 ppm
Sep 26 15:00:30 dist-test-slave-jnhs krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434830, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:30.123282 12365 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:00:30.123875 12365 webserver.cc:460] Webserver started at http://127.8.102.2:40133/ using document root <none> and password file <none>
I20260926 15:00:30.124145 12365 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:30.124194 12365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:30.124333 12365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:30.125433 12365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "e75ca423803c4523a0d3c6edbd286e01"
format_stamp: "Formatted at 2026-09-26 15:00:30 on dist-test-slave-jnhs"
server_key: "4dee4b9be2dffa7eb3d0e1f1411814ad"
server_key_iv: "cbcf2415a134584b1ab4dd96533fc318"
server_key_version: "encryptionkey@0"
I20260926 15:00:30.125804 12365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "e75ca423803c4523a0d3c6edbd286e01"
format_stamp: "Formatted at 2026-09-26 15:00:30 on dist-test-slave-jnhs"
server_key: "4dee4b9be2dffa7eb3d0e1f1411814ad"
server_key_iv: "cbcf2415a134584b1ab4dd96533fc318"
server_key_version: "encryptionkey@0"
I20260926 15:00:30.127372 12365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:00:30.128274 12380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:30.128482 12365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:30.128579 12365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "e75ca423803c4523a0d3c6edbd286e01"
format_stamp: "Formatted at 2026-09-26 15:00:30 on dist-test-slave-jnhs"
server_key: "4dee4b9be2dffa7eb3d0e1f1411814ad"
server_key_iv: "cbcf2415a134584b1ab4dd96533fc318"
server_key_version: "encryptionkey@0"
I20260926 15:00:30.128641 12365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:30.141422 12365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:30.143615 12365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:30.143738 12365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:30.143929 12365 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:30.144243 12365 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:30.144275 12365 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:30.144299 12365 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:30.144311 12365 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:30.150122 12365 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:37521
I20260926 15:00:30.150193 12493 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:37521 every 8 connection(s)
I20260926 15:00:30.150523 12365 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
Sep 26 15:00:30 dist-test-slave-jnhs krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434830, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:30.158156 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 12365
I20260926 15:00:30.158286 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:00:30.158471 8600 external_mini_cluster.cc:1428] Setting key 67c461b1c8f5d05499facbdb6b323e87
I20260926 15:00:30.160898 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:30.151294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:39665 (local address 127.8.102.62:39647)
0926 15:00:30.151367 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:30.151369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:30.151567 (+ 198us) server_negotiation.cc:407] Connection header received
0926 15:00:30.152583 (+ 1016us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:30.152586 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:30.152598 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:30.152654 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:30.153288 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:30.153795 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:30.154911 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:30.155109 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:30.155218 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:30.157504 (+ 2286us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:30.157505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:30.157506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:30.157513 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:30.159245 (+ 1732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:30.159743 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:30.159744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:30.159744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:30.159800 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:30.160122 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:30.160123 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:30.160123 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:30.160241 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:30.160297 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:30.160641 (+ 344us) server_negotiation.cc:299] Negotiation successful
0926 15:00:30.160774 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 15:00:30.161113 12496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:30.151269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:39647 (local address 127.8.102.2:39665)
0926 15:00:30.151441 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:30.151477 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:30.152454 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:30.152734 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:30.152738 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:30.152994 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:30.153184 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:30.153188 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:30.153914 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:30.153915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:30.154817 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:30.154819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:30.155225 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:30.155226 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:30.155370 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:30.155938 (+ 568us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:30.155946 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:30.157395 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:30.159312 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:30.159320 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:30.159323 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:30.159661 (+ 338us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:30.159889 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:30.159889 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:30.159890 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:30.160052 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:30.160310 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:30.160316 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:30.160496 (+ 180us) client_negotiation.cc:769] Sending connection context
0926 15:00:30.160614 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 15:00:30.160813 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":74,"threads_started":1}
I20260926 15:00:30.161728 12494 heartbeater.cc:347] Connected to a master server at 127.8.102.62:39647
I20260926 15:00:30.161850 12494 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:30.162091 12494 heartbeater.cc:511] Master 127.8.102.62:39647 requested a full tablet report, sending...
I20260926 15:00:30.162564 12169 ts_manager.cc:194] Registered new tserver with Master: e75ca423803c4523a0d3c6edbd286e01 (127.8.102.2:37521)
I20260926 15:00:30.162971 12169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:39665
WARNING: no policy specified for kudu/127.8.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
I20260926 15:00:30.186823 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:39647
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36463
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:30.268146 12499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:30.268402 12499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:30.268438 12499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:30.269716 12499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:30.269791 12499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:30.269837 12499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:00:30.271384 12499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36463
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:39647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:30.271662 12499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:30.271901 12499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:30.277283 12505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:30.277283 12507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:30.277303 12504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:30.277711 12499 server_base.cc:1034] running on GCE node
I20260926 15:00:30.277957 12499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:30.278204 12499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:30.279351 12499 hybrid_clock.cc:648] HybridClock initialized: now 1790434830279337 us; error 30 us; skew 500 ppm
Sep 26 15:00:30 dist-test-slave-jnhs krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434830, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:30.281908 12499 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:00:30.282378 12499 webserver.cc:460] Webserver started at http://127.8.102.3:36351/ using document root <none> and password file <none>
I20260926 15:00:30.282572 12499 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:30.282613 12499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:30.282764 12499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:30.283756 12499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "d309a68ad8934f3dbe090716a11e60c2"
format_stamp: "Formatted at 2026-09-26 15:00:30 on dist-test-slave-jnhs"
server_key: "cbe987eb2ed2c74ba11962b217a7dd61"
server_key_iv: "fb2f478bbdb066a57f9a4d438ca32d36"
server_key_version: "encryptionkey@0"
I20260926 15:00:30.284046 12499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "d309a68ad8934f3dbe090716a11e60c2"
format_stamp: "Formatted at 2026-09-26 15:00:30 on dist-test-slave-jnhs"
server_key: "cbe987eb2ed2c74ba11962b217a7dd61"
server_key_iv: "fb2f478bbdb066a57f9a4d438ca32d36"
server_key_version: "encryptionkey@0"
I20260926 15:00:30.285285 12499 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:00:30.286010 12514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:30.286195 12499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:30.286298 12499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "d309a68ad8934f3dbe090716a11e60c2"
format_stamp: "Formatted at 2026-09-26 15:00:30 on dist-test-slave-jnhs"
server_key: "cbe987eb2ed2c74ba11962b217a7dd61"
server_key_iv: "fb2f478bbdb066a57f9a4d438ca32d36"
server_key_version: "encryptionkey@0"
I20260926 15:00:30.286376 12499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:30.309876 12499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:30.312050 12499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:30.312240 12499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:30.312482 12499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:30.312791 12499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:30.312845 12499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:30.312891 12499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:30.312920 12499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:30.318866 12499 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:34491
I20260926 15:00:30.318919 12627 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:34491 every 8 connection(s)
I20260926 15:00:30.319280 12499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:00:30.322258 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 12499
I20260926 15:00:30.322357 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:00:30.322542 8600 external_mini_cluster.cc:1428] Setting key e1c3adc104f8ed618b3348983d8df74b
Sep 26 15:00:30 dist-test-slave-jnhs krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434830, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:30.329428 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:30.320066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:46709 (local address 127.8.102.62:39647)
0926 15:00:30.320115 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:30.320117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:30.320293 (+ 176us) server_negotiation.cc:407] Connection header received
0926 15:00:30.321249 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:30.321251 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:30.321262 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:30.321327 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:30.322067 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:30.322915 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:30.324086 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:30.324277 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:30.324334 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:30.326660 (+ 2326us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:30.326662 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:30.326662 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:30.326670 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:30.328009 (+ 1339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:30.328437 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:30.328438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:30.328439 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:30.328501 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:30.328762 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:30.328763 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:30.328763 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:30.328886 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:30.328961 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:30.329240 (+ 279us) server_negotiation.cc:299] Negotiation successful
0926 15:00:30.329322 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 15:00:30.329648 12630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:30.320013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:39647 (local address 127.8.102.3:46709)
0926 15:00:30.320174 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:30.320215 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:30.321139 (+ 924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:30.321406 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:30.321410 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:30.321714 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:30.321918 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:30.321923 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:30.323004 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:30.323005 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:30.324005 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:30.324007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:30.324350 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:30.324350 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:30.324453 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:30.324974 (+ 521us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:30.324986 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:30.326484 (+ 1498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:30.328101 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:30.328108 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:30.328110 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:30.328345 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:30.328567 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:30.328568 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:30.328569 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:30.328685 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:30.328951 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:30.328957 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:30.329104 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 15:00:30.329207 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 15:00:30.329391 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":92,"threads_started":1}
I20260926 15:00:30.330210 12628 heartbeater.cc:347] Connected to a master server at 127.8.102.62:39647
I20260926 15:00:30.330343 12628 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:30.330587 12628 heartbeater.cc:511] Master 127.8.102.62:39647 requested a full tablet report, sending...
I20260926 15:00:30.331029 12169 ts_manager.cc:194] Registered new tserver with Master: d309a68ad8934f3dbe090716a11e60c2 (127.8.102.3:34491)
I20260926 15:00:30.331447 12169 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:46709
I20260926 15:00:30.334281 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 26 15:00:30 dist-test-slave-jnhs krb5kdc[12124](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 15:00:30 dist-test-slave-jnhs krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.1@KRBTEST.COM
I20260926 15:00:30.357177 12635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:30.349376 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45456 (local address 127.8.102.1:37521)
0926 15:00:30.349594 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:30.349601 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:30.349620 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:00:30.349648 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:30.349650 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:30.349665 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:30.349733 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:30.350279 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:30.351105 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:30.351854 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:30.352200 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:30.352260 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:30.354498 (+ 2238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:30.354506 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:30.354506 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:30.354513 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:30.356068 (+ 1555us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:30.356428 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:30.356431 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:30.356431 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:30.356490 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:30.356653 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:30.356653 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:30.356653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:30.356783 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:30.356884 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:30.356947 (+ 63us) server_negotiation.cc:299] Negotiation successful
0926 15:00:30.357058 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":144,"threads_started":1}
Sep 26 15:00:30 dist-test-slave-jnhs krb5kdc[12124](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434830, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:30.365299 12208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:30.358467 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43970 (local address 127.8.102.62:39647)
0926 15:00:30.358512 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:30.358514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:30.358545 (+ 31us) server_negotiation.cc:407] Connection header received
0926 15:00:30.358619 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:30.358620 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:30.358629 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:30.358732 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:30.359207 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:30.359704 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:30.360353 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:30.360590 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:30.360653 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:30.362986 (+ 2333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:30.362987 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:30.362987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:30.362995 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:30.364192 (+ 1197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:30.364524 (+ 332us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:30.364525 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:30.364525 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:30.364571 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:30.364806 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:30.364807 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:30.364807 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:30.364967 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:30.365048 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:30.365138 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 15:00:30.365202 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
W20260926 15:00:30.365587 12169 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43970
I20260926 15:00:30.365993 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 12231
I20260926 15:00:30.370352 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 12365
I20260926 15:00:30.374614 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 12499
I20260926 15:00:30.379026 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 12137
2026-09-26T15:00:30Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1872 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:00:30 dist-test-slave-jnhs krb5kdc[12640](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:00:30 dist-test-slave-jnhs krb5kdc[12640](info): set up 2 sockets
Sep 26 15:00:30 dist-test-slave-jnhs krb5kdc[12640](info): commencing operation
krb5kdc: starting...
W20260926 15:00:31.418958 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.020s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:00:31 dist-test-slave-jnhs krb5kdc[12640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:00:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:00:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:00:31.547837 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:37767
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:37767
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:31.629338 12653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:31.629611 12653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:31.629650 12653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:31.630861 12653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:31.630939 12653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:31.630962 12653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:31.630985 12653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:31.631014 12653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:00:31.632565 12653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:37767
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:37767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12653
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:31.632849 12653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:31.633095 12653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:31.637989 12658 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:31.637988 12661 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:31.638248 12653 server_base.cc:1034] running on GCE node
W20260926 15:00:31.638015 12659 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:31.638561 12653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:31.638924 12653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:31.640101 12653 hybrid_clock.cc:648] HybridClock initialized: now 1790434831640078 us; error 40 us; skew 500 ppm
Sep 26 15:00:31 dist-test-slave-jnhs krb5kdc[12640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434831, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:31.649401 12653 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:00:31.649837 12653 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:31.649892 12653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:31.650059 12653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:31.651250 12653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "60fb11cfccb341dda185cc07c73dd885"
format_stamp: "Formatted at 2026-09-26 15:00:31 on dist-test-slave-jnhs"
server_key: "7f1d50d89e54896d6cf50bafdaa2a5a4"
server_key_iv: "2e531bce6f3f625c03f35dedf3cb9b5f"
server_key_version: "encryptionkey@0"
I20260926 15:00:31.651618 12653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "60fb11cfccb341dda185cc07c73dd885"
format_stamp: "Formatted at 2026-09-26 15:00:31 on dist-test-slave-jnhs"
server_key: "7f1d50d89e54896d6cf50bafdaa2a5a4"
server_key_iv: "2e531bce6f3f625c03f35dedf3cb9b5f"
server_key_version: "encryptionkey@0"
I20260926 15:00:31.652990 12653 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:00:31.653820 12667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:31.654050 12653 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:31.654161 12653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "60fb11cfccb341dda185cc07c73dd885"
format_stamp: "Formatted at 2026-09-26 15:00:31 on dist-test-slave-jnhs"
server_key: "7f1d50d89e54896d6cf50bafdaa2a5a4"
server_key_iv: "2e531bce6f3f625c03f35dedf3cb9b5f"
server_key_version: "encryptionkey@0"
I20260926 15:00:31.654239 12653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:31.664642 12653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:31.666791 12653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:31.666963 12653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:31.670809 12653 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:37767
I20260926 15:00:31.670883 12719 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:37767 every 8 connection(s)
I20260926 15:00:31.671186 12653 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:00:31.672242 12720 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:31.673123 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 12653
I20260926 15:00:31.673255 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:00:31.673439 8600 external_mini_cluster.cc:1428] Setting key 55377af2b47ea34746df2185f0888f8e
I20260926 15:00:31.675791 12720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885: Bootstrap starting.
I20260926 15:00:31.676656 12720 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:31.676976 12720 log.cc:824] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:31.677803 12720 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885: No bootstrap required, opened a new log
I20260926 15:00:31.680238 12720 raft_consensus.cc:348] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60fb11cfccb341dda185cc07c73dd885" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37767 } }
I20260926 15:00:31.680423 12720 raft_consensus.cc:374] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:31.680474 12720 raft_consensus.cc:729] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 60fb11cfccb341dda185cc07c73dd885, State: Initialized, Role: FOLLOWER
I20260926 15:00:31.680681 12720 consensus_queue.cc:260] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60fb11cfccb341dda185cc07c73dd885" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37767 } }
I20260926 15:00:31.680787 12720 raft_consensus.cc:388] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:00:31.680833 12720 raft_consensus.cc:482] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:00:31.680891 12720 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885 [term 0 FOLLOWER]: Advancing to term 1
Sep 26 15:00:31 dist-test-slave-jnhs krb5kdc[12640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:31.681607 12720 raft_consensus.cc:504] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60fb11cfccb341dda185cc07c73dd885" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37767 } }
I20260926 15:00:31.681747 12720 leader_election.cc:302] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 60fb11cfccb341dda185cc07c73dd885; no voters:
I20260926 15:00:31.681939 12720 leader_election.cc:288] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:00:31.682024 12725 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:31.682166 12725 raft_consensus.cc:686] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885 [term 1 LEADER]: Becoming Leader. State: Replica: 60fb11cfccb341dda185cc07c73dd885, State: Running, Role: LEADER
I20260926 15:00:31.682266 12725 consensus_queue.cc:237] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60fb11cfccb341dda185cc07c73dd885" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37767 } }
I20260926 15:00:31.682296 12720 sys_catalog.cc:564] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:31.682742 12726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "60fb11cfccb341dda185cc07c73dd885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60fb11cfccb341dda185cc07c73dd885" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37767 } } }
I20260926 15:00:31.682870 12726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:31.682734 12727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 60fb11cfccb341dda185cc07c73dd885. Latest consensus state: current_term: 1 leader_uuid: "60fb11cfccb341dda185cc07c73dd885" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "60fb11cfccb341dda185cc07c73dd885" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37767 } } }
I20260926 15:00:31.683154 12727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:31.684155 12732 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:00:31.685658 12732 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:00:31.686606 12723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:31.674034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54924 (local address 127.8.102.62:37767)
0926 15:00:31.674244 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:31.674256 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:31.674300 (+ 44us) server_negotiation.cc:407] Connection header received
0926 15:00:31.674952 (+ 652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:31.674964 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:31.675037 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:31.675213 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:31.675893 (+ 680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:31.677418 (+ 1525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:31.678330 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:31.679059 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:31.679190 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:31.681744 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:31.681748 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:31.681749 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:31.681763 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:31.684719 (+ 2956us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:31.685224 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:31.685227 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:31.685228 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:31.685350 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:31.685644 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:31.685645 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:31.685646 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:31.685969 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:31.686133 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:31.686255 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 15:00:31.686403 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":90,"threads_started":1}
I20260926 15:00:31.687484 12732 catalog_manager.cc:1347] Generated new cluster ID: c37d77ed435f417b8dbfabbce75ac75b
I20260926 15:00:31.687526 12732 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:00:31.709390 12732 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:00:31.710211 12732 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:00:31.717607 12732 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885: Generated new TSK 0
I20260926 15:00:31.717818 12732 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
I20260926 15:00:31.754788 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37767
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:31.841814 12746 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:31.842060 12746 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:31.842104 12746 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:31.843461 12746 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:31.843546 12746 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:31.843595 12746 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:00:31.845201 12746 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12746
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:31.845494 12746 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:31.845747 12746 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:31.850915 12751 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:31.850916 12754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:31.850916 12752 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:31.851269 12746 server_base.cc:1034] running on GCE node
I20260926 15:00:31.851573 12746 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:31.851843 12746 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:31.853009 12746 hybrid_clock.cc:648] HybridClock initialized: now 1790434831852993 us; error 46 us; skew 500 ppm
Sep 26 15:00:31 dist-test-slave-jnhs krb5kdc[12640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434831, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:31.855985 12746 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:00:31.856477 12746 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:31.856536 12746 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:31.856633 12746 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:31.857748 12746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "3bea16947c3245129943a248f3bcc222"
format_stamp: "Formatted at 2026-09-26 15:00:31 on dist-test-slave-jnhs"
server_key: "3c3790c0de4bf8269743a888998136cb"
server_key_iv: "ba397aa39dcede91704581d1ec370646"
server_key_version: "encryptionkey@0"
I20260926 15:00:31.858160 12746 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "3bea16947c3245129943a248f3bcc222"
format_stamp: "Formatted at 2026-09-26 15:00:31 on dist-test-slave-jnhs"
server_key: "3c3790c0de4bf8269743a888998136cb"
server_key_iv: "ba397aa39dcede91704581d1ec370646"
server_key_version: "encryptionkey@0"
I20260926 15:00:31.859649 12746 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 15:00:31.860525 12760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:31.860715 12746 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 15:00:31.860826 12746 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "3bea16947c3245129943a248f3bcc222"
format_stamp: "Formatted at 2026-09-26 15:00:31 on dist-test-slave-jnhs"
server_key: "3c3790c0de4bf8269743a888998136cb"
server_key_iv: "ba397aa39dcede91704581d1ec370646"
server_key_version: "encryptionkey@0"
I20260926 15:00:31.860905 12746 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:31.874734 12746 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:31.877125 12746 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:31.877327 12746 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:31.877648 12746 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:31.877983 12746 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:31.878041 12746 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:31.878083 12746 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:31.878108 12746 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:31.884608 12746 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:36943
I20260926 15:00:31.884639 12873 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:36943 every 8 connection(s)
I20260926 15:00:31.884948 12746 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:00:31.890318 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 12746
I20260926 15:00:31.890465 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:00:31.890697 8600 external_mini_cluster.cc:1428] Setting key 161dbaeaf461d20cbd6982a2b3ab1ce1
Sep 26 15:00:31 dist-test-slave-jnhs krb5kdc[12640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434831, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:31.896170 12723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:31.885763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:33043 (local address 127.8.102.62:37767)
0926 15:00:31.885862 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:31.885865 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:31.886085 (+ 220us) server_negotiation.cc:407] Connection header received
0926 15:00:31.886586 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:31.886587 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:31.886600 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:31.886745 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:31.887726 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:31.888250 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:31.889363 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:31.889587 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:31.889657 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:31.892376 (+ 2719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:31.892377 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:31.892378 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:31.892386 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:31.894230 (+ 1844us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:31.894786 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:31.894788 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:31.894788 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:31.894859 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:31.895166 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:31.895167 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:31.895168 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:31.895334 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:31.895420 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:31.895933 (+ 513us) server_negotiation.cc:299] Negotiation successful
0926 15:00:31.896058 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260926 15:00:31.896256 12876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:31.885749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:37767 (local address 127.8.102.1:33043)
0926 15:00:31.885916 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:31.885969 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:31.886462 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:31.886828 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:31.886832 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:31.887119 (+ 287us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:31.887609 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:31.887615 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:31.888364 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:31.888365 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:31.889254 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:31.889256 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:31.889678 (+ 422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:31.889679 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:31.889786 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:31.890423 (+ 637us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:31.890432 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:31.892248 (+ 1816us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:31.894356 (+ 2108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:31.894363 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:31.894372 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:31.894695 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:31.894952 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:31.894953 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:31.894953 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:31.895083 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:31.895460 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:31.895469 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:31.895721 (+ 252us) client_negotiation.cc:769] Sending connection context
0926 15:00:31.895863 (+ 142us) client_negotiation.cc:240] Negotiation successful
0926 15:00:31.896016 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":88,"threads_started":1}
I20260926 15:00:31.896904 12874 heartbeater.cc:347] Connected to a master server at 127.8.102.62:37767
I20260926 15:00:31.897053 12874 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:31.897253 12874 heartbeater.cc:511] Master 127.8.102.62:37767 requested a full tablet report, sending...
I20260926 15:00:31.897811 12684 ts_manager.cc:194] Registered new tserver with Master: 3bea16947c3245129943a248f3bcc222 (127.8.102.1:36943)
I20260926 15:00:31.898448 12684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:33043
WARNING: no policy specified for kudu/127.8.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
I20260926 15:00:31.922888 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37767
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:32.009256 12879 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:32.009488 12879 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:32.009516 12879 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:32.010973 12879 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:32.011019 12879 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:32.011050 12879 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:00:32.012828 12879 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.12879
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:32.013123 12879 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:32.013392 12879 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 15:00:32.018787 12879 server_base.cc:1034] running on GCE node
W20260926 15:00:32.018712 12884 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:32.018761 12887 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:32.018712 12885 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:32.019326 12879 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:32.019600 12879 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:32.020803 12879 hybrid_clock.cc:648] HybridClock initialized: now 1790434832020768 us; error 48 us; skew 500 ppm
Sep 26 15:00:32 dist-test-slave-jnhs krb5kdc[12640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434832, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:32.023717 12879 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:00:32.024118 12879 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:32.024171 12879 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:32.024310 12879 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:32.025460 12879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "9e3f8a48ffc44feba530e54fac654770"
format_stamp: "Formatted at 2026-09-26 15:00:32 on dist-test-slave-jnhs"
server_key: "aeecbb01b5982d157b47a715c3535212"
server_key_iv: "4ca16cfc087232f54490e93952813228"
server_key_version: "encryptionkey@0"
I20260926 15:00:32.025837 12879 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "9e3f8a48ffc44feba530e54fac654770"
format_stamp: "Formatted at 2026-09-26 15:00:32 on dist-test-slave-jnhs"
server_key: "aeecbb01b5982d157b47a715c3535212"
server_key_iv: "4ca16cfc087232f54490e93952813228"
server_key_version: "encryptionkey@0"
I20260926 15:00:32.027341 12879 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:00:32.028204 12893 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:32.028555 12879 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:32.028646 12879 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "9e3f8a48ffc44feba530e54fac654770"
format_stamp: "Formatted at 2026-09-26 15:00:32 on dist-test-slave-jnhs"
server_key: "aeecbb01b5982d157b47a715c3535212"
server_key_iv: "4ca16cfc087232f54490e93952813228"
server_key_version: "encryptionkey@0"
I20260926 15:00:32.028759 12879 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:32.039052 12879 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:32.041472 12879 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:32.041664 12879 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:32.041949 12879 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:32.042295 12879 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:32.042330 12879 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:32.042354 12879 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:32.042393 12879 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:32.048543 12879 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:42309
I20260926 15:00:32.048620 13006 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:42309 every 8 connection(s)
I20260926 15:00:32.048941 12879 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
Sep 26 15:00:32 dist-test-slave-jnhs krb5kdc[12640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434832, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:32.058398 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 12879
I20260926 15:00:32.058514 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:00:32.058745 8600 external_mini_cluster.cc:1428] Setting key 84c6912b9fb2073f516d8d3fe9797838
I20260926 15:00:32.059826 12723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.049777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:52009 (local address 127.8.102.62:37767)
0926 15:00:32.049887 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:32.049889 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:32.050107 (+ 218us) server_negotiation.cc:407] Connection header received
0926 15:00:32.050579 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:32.050580 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:32.050593 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:32.050808 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:32.051850 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.052374 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.053446 (+ 1072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.053658 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.053730 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:32.056122 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:32.056123 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:32.056123 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:32.056131 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:32.057799 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:32.058403 (+ 604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:32.058405 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:32.058405 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:32.058483 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:32.058961 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:32.058962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:32.058962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:32.059088 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:32.059140 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:32.059579 (+ 439us) server_negotiation.cc:299] Negotiation successful
0926 15:00:32.059689 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 15:00:32.059847 13009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.049746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:37767 (local address 127.8.102.2:52009)
0926 15:00:32.049912 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:32.049972 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:32.050459 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:32.050877 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:32.050880 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:32.051160 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:32.051713 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:32.051719 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.052473 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.052474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:32.053335 (+ 861us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:32.053337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.053763 (+ 426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.053764 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:32.053892 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:32.054524 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:32.054533 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:32.056026 (+ 1493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:32.057994 (+ 1968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:32.058008 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:32.058010 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:32.058318 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:32.058702 (+ 384us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:32.058703 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:32.058703 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:32.058854 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:32.059150 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:32.059157 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:32.059326 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 15:00:32.059506 (+ 180us) client_negotiation.cc:240] Negotiation successful
0926 15:00:32.059668 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260926 15:00:32.060508 13007 heartbeater.cc:347] Connected to a master server at 127.8.102.62:37767
I20260926 15:00:32.060626 13007 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:32.060803 13007 heartbeater.cc:511] Master 127.8.102.62:37767 requested a full tablet report, sending...
I20260926 15:00:32.061262 12684 ts_manager.cc:194] Registered new tserver with Master: 9e3f8a48ffc44feba530e54fac654770 (127.8.102.2:42309)
I20260926 15:00:32.061650 12684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:52009
WARNING: no policy specified for kudu/127.8.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
I20260926 15:00:32.090471 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37767
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36615
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:32.176616 13012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:32.176873 13012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:32.176916 13012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:32.178150 13012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:32.178224 13012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:32.178270 13012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:00:32.179930 13012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36615
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37767
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13012
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:32.180230 13012 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:32.180477 13012 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:32.186103 13020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:32.186100 13017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:32.186156 13018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:32.186493 13012 server_base.cc:1034] running on GCE node
I20260926 15:00:32.186789 13012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:32.187067 13012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:32.188233 13012 hybrid_clock.cc:648] HybridClock initialized: now 1790434832188221 us; error 32 us; skew 500 ppm
Sep 26 15:00:32 dist-test-slave-jnhs krb5kdc[12640](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434832, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:32.191042 13012 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:00:32.191421 13012 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:32.191473 13012 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:32.191640 13012 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:32.192754 13012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "47b24d1dc7e44371bbfe102ed27f2984"
format_stamp: "Formatted at 2026-09-26 15:00:32 on dist-test-slave-jnhs"
server_key: "49f92291e2b7a94838288c53d432f28b"
server_key_iv: "25fdb4e3d2f6fe509d0cc4d1cd5f6919"
server_key_version: "encryptionkey@0"
I20260926 15:00:32.193137 13012 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "47b24d1dc7e44371bbfe102ed27f2984"
format_stamp: "Formatted at 2026-09-26 15:00:32 on dist-test-slave-jnhs"
server_key: "49f92291e2b7a94838288c53d432f28b"
server_key_iv: "25fdb4e3d2f6fe509d0cc4d1cd5f6919"
server_key_version: "encryptionkey@0"
I20260926 15:00:32.194561 13012 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:00:32.195430 13026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:32.195650 13012 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:32.195727 13012 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "47b24d1dc7e44371bbfe102ed27f2984"
format_stamp: "Formatted at 2026-09-26 15:00:32 on dist-test-slave-jnhs"
server_key: "49f92291e2b7a94838288c53d432f28b"
server_key_iv: "25fdb4e3d2f6fe509d0cc4d1cd5f6919"
server_key_version: "encryptionkey@0"
I20260926 15:00:32.195827 13012 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:32.206930 13012 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:32.209686 13012 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:32.209839 13012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:32.210250 13012 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:32.210711 13012 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:32.210781 13012 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:32.210829 13012 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:32.210881 13012 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:32.217455 13012 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:33993
I20260926 15:00:32.217514 13139 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:33993 every 8 connection(s)
I20260926 15:00:32.217857 13012 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
Sep 26 15:00:32 dist-test-slave-jnhs krb5kdc[12640](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434832, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:32.226290 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 13012
I20260926 15:00:32.226429 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:00:32.226673 8600 external_mini_cluster.cc:1428] Setting key 63d308bbc89d83621202a679fe18d8a1
I20260926 15:00:32.228907 12723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.218719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:35363 (local address 127.8.102.62:37767)
0926 15:00:32.218796 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:32.218799 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:32.219149 (+ 350us) server_negotiation.cc:407] Connection header received
0926 15:00:32.219729 (+ 580us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:32.219730 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:32.219742 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:32.219799 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:32.220883 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.221393 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.222512 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.222804 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.222873 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:32.225290 (+ 2417us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:32.225292 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:32.225292 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:32.225300 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:32.227322 (+ 2022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:32.227845 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:32.227846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:32.227847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:32.227928 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:32.228230 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:32.228230 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:32.228231 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:32.228348 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:32.228413 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:32.228741 (+ 328us) server_negotiation.cc:299] Negotiation successful
0926 15:00:32.228822 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 15:00:32.229007 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.218704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:37767 (local address 127.8.102.3:35363)
0926 15:00:32.218984 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:32.219027 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:32.219537 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:32.219894 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:32.219897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:32.220186 (+ 289us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:32.220743 (+ 557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:32.220749 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.221537 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.221537 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:32.222352 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:32.222354 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.222916 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.222918 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:32.223073 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:32.223617 (+ 544us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:32.223627 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:32.225175 (+ 1548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:32.227412 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:32.227419 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:32.227422 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:32.227750 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:32.228024 (+ 274us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:32.228025 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:32.228026 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:32.228157 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:32.228437 (+ 280us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:32.228443 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:32.228579 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 15:00:32.228681 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 15:00:32.228834 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":235,"thread_start_us":99,"threads_started":1}
I20260926 15:00:32.229526 13140 heartbeater.cc:347] Connected to a master server at 127.8.102.62:37767
I20260926 15:00:32.229630 13140 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:32.229754 13140 heartbeater.cc:511] Master 127.8.102.62:37767 requested a full tablet report, sending...
I20260926 15:00:32.230104 12684 ts_manager.cc:194] Registered new tserver with Master: 47b24d1dc7e44371bbfe102ed27f2984 (127.8.102.3:33993)
I20260926 15:00:32.230432 12684 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:35363
I20260926 15:00:32.238847 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:00:32.246516 12723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.240219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54938 (local address 127.8.102.62:37767)
0926 15:00:32.240296 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:32.240298 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:32.240322 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:00:32.240378 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:32.240378 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:32.240386 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:32.240434 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:32.240983 (+ 549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.241460 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.242179 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.242358 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.242410 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:32.244003 (+ 1593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:32.244005 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:32.244005 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:32.244012 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:32.245484 (+ 1472us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:32.245850 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:32.245851 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:32.245851 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:32.245901 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:32.246076 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:32.246076 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:32.246077 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:32.246217 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:32.246293 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:32.246394 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 15:00:32.246451 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 15:00:32.248529 12684 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54938:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:00:32.250173 12684 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:00:32.257233 13151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.252957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:33993 (local address 127.0.0.1:39222)
0926 15:00:32.253224 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:32.253269 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:32.253375 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:32.253691 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:32.253695 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:32.253698 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:32.253768 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:32.253770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.255099 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.255100 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:32.255880 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:32.255882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.256855 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.256856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:32.256956 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:32.256970 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 15:00:32.257041 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 15:00:32.257104 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":52,"threads_started":1}
I20260926 15:00:32.257229 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.253114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39222 (local address 127.8.102.3:33993)
0926 15:00:32.253280 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:32.253285 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:32.253333 (+ 48us) server_negotiation.cc:407] Connection header received
0926 15:00:32.253485 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:32.253488 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:32.253506 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:32.253556 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:32.253863 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.254988 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.255977 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.256695 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.256763 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:32.256822 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:32.257083 (+ 261us) server_negotiation.cc:299] Negotiation successful
0926 15:00:32.257136 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
I20260926 15:00:32.257378 13152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.253164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:36943 (local address 127.0.0.1:36680)
0926 15:00:32.253389 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:32.253401 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:32.253456 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:32.253754 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:32.253755 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:32.253757 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:32.253807 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:32.253809 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.254855 (+ 1046us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.254857 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:32.256165 (+ 1308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:32.256168 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.257148 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.257149 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:32.257212 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:32.257217 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:00:32.257249 (+ 32us) client_negotiation.cc:240] Negotiation successful
0926 15:00:32.257295 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":203,"thread_start_us":60,"threads_started":1}
I20260926 15:00:32.258065 13156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.253371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36680 (local address 127.8.102.1:36943)
0926 15:00:32.253611 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:32.253616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:32.253625 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:32.253663 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:32.253665 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:32.253679 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:32.253756 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:32.253866 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.254769 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.256282 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.257052 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.257701 (+ 649us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:32.257773 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:32.257798 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 15:00:32.257934 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":69,"threads_started":1}
I20260926 15:00:32.258081 13074 tablet_service.cc:1467] Processing CreateTablet for tablet f9b64366e12c479aa8a17b9119e066bd (DEFAULT_TABLE table=test-table [id=e90d3ae8bdf84614b2ffba79ddaed866]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:32.258488 13074 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b64366e12c479aa8a17b9119e066bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:32.258826 12808 tablet_service.cc:1467] Processing CreateTablet for tablet f9b64366e12c479aa8a17b9119e066bd (DEFAULT_TABLE table=test-table [id=e90d3ae8bdf84614b2ffba79ddaed866]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:32.259100 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.252956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:42309 (local address 127.0.0.1:36216)
0926 15:00:32.253327 (+ 371us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:32.253337 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:32.254574 (+ 1237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:32.254978 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:32.254979 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:32.254981 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:32.255046 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:32.255048 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.256303 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.256304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:32.257342 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:32.257345 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.258765 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.258767 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:32.258864 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:32.258867 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:00:32.258950 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 15:00:32.259009 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":354,"mutex_wait_us":5,"thread_start_us":95,"threads_started":1}
I20260926 15:00:32.259146 12808 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b64366e12c479aa8a17b9119e066bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:32.259438 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.253039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36216 (local address 127.8.102.2:42309)
0926 15:00:32.253196 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:32.253201 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:32.254539 (+ 1338us) server_negotiation.cc:407] Connection header received
0926 15:00:32.254775 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:32.254778 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:32.254793 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:32.254843 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:32.255145 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.256203 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.257637 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.258563 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.258713 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:32.258776 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:32.259313 (+ 537us) server_negotiation.cc:299] Negotiation successful
0926 15:00:32.259374 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":91,"threads_started":1}
I20260926 15:00:32.262422 13157 tablet_bootstrap.cc:492] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984: Bootstrap starting.
I20260926 15:00:32.262809 13158 tablet_bootstrap.cc:492] T f9b64366e12c479aa8a17b9119e066bd P 3bea16947c3245129943a248f3bcc222: Bootstrap starting.
I20260926 15:00:32.262805 12941 tablet_service.cc:1467] Processing CreateTablet for tablet f9b64366e12c479aa8a17b9119e066bd (DEFAULT_TABLE table=test-table [id=e90d3ae8bdf84614b2ffba79ddaed866]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:32.263149 12941 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f9b64366e12c479aa8a17b9119e066bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:32.263149 13157 tablet_bootstrap.cc:654] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:32.263494 13158 tablet_bootstrap.cc:654] T f9b64366e12c479aa8a17b9119e066bd P 3bea16947c3245129943a248f3bcc222: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:32.263810 13157 log.cc:824] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:32.264103 13158 log.cc:824] T f9b64366e12c479aa8a17b9119e066bd P 3bea16947c3245129943a248f3bcc222: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:32.264658 13157 tablet_bootstrap.cc:492] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984: No bootstrap required, opened a new log
I20260926 15:00:32.264809 13157 ts_tablet_manager.cc:1397] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:00:32.264858 13158 tablet_bootstrap.cc:492] T f9b64366e12c479aa8a17b9119e066bd P 3bea16947c3245129943a248f3bcc222: No bootstrap required, opened a new log
I20260926 15:00:32.264905 13158 ts_tablet_manager.cc:1397] T f9b64366e12c479aa8a17b9119e066bd P 3bea16947c3245129943a248f3bcc222: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:00:32.266237 13158 raft_consensus.cc:348] T f9b64366e12c479aa8a17b9119e066bd P 3bea16947c3245129943a248f3bcc222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3f8a48ffc44feba530e54fac654770" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 42309 } } peers { permanent_uuid: "3bea16947c3245129943a248f3bcc222" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36943 } } peers { permanent_uuid: "47b24d1dc7e44371bbfe102ed27f2984" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 33993 } }
I20260926 15:00:32.266381 13158 raft_consensus.cc:374] T f9b64366e12c479aa8a17b9119e066bd P 3bea16947c3245129943a248f3bcc222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:32.266404 13158 raft_consensus.cc:729] T f9b64366e12c479aa8a17b9119e066bd P 3bea16947c3245129943a248f3bcc222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3bea16947c3245129943a248f3bcc222, State: Initialized, Role: FOLLOWER
I20260926 15:00:32.266528 13158 consensus_queue.cc:260] T f9b64366e12c479aa8a17b9119e066bd P 3bea16947c3245129943a248f3bcc222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3f8a48ffc44feba530e54fac654770" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 42309 } } peers { permanent_uuid: "3bea16947c3245129943a248f3bcc222" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36943 } } peers { permanent_uuid: "47b24d1dc7e44371bbfe102ed27f2984" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 33993 } }
I20260926 15:00:32.266803 13161 tablet_bootstrap.cc:492] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770: Bootstrap starting.
I20260926 15:00:32.266970 13158 ts_tablet_manager.cc:1428] T f9b64366e12c479aa8a17b9119e066bd P 3bea16947c3245129943a248f3bcc222: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 15:00:32.267102 12874 heartbeater.cc:503] Master 127.8.102.62:37767 was elected leader, sending a full tablet report...
I20260926 15:00:32.267494 13161 tablet_bootstrap.cc:654] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:32.267830 13157 raft_consensus.cc:348] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3f8a48ffc44feba530e54fac654770" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 42309 } } peers { permanent_uuid: "3bea16947c3245129943a248f3bcc222" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36943 } } peers { permanent_uuid: "47b24d1dc7e44371bbfe102ed27f2984" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 33993 } }
I20260926 15:00:32.268018 13157 raft_consensus.cc:374] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:32.268064 13157 raft_consensus.cc:729] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47b24d1dc7e44371bbfe102ed27f2984, State: Initialized, Role: FOLLOWER
I20260926 15:00:32.268111 13161 log.cc:824] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:32.268182 13157 consensus_queue.cc:260] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3f8a48ffc44feba530e54fac654770" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 42309 } } peers { permanent_uuid: "3bea16947c3245129943a248f3bcc222" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36943 } } peers { permanent_uuid: "47b24d1dc7e44371bbfe102ed27f2984" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 33993 } }
I20260926 15:00:32.268481 13140 heartbeater.cc:503] Master 127.8.102.62:37767 was elected leader, sending a full tablet report...
I20260926 15:00:32.268481 13157 ts_tablet_manager.cc:1428] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 15:00:32.268960 13161 tablet_bootstrap.cc:492] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770: No bootstrap required, opened a new log
I20260926 15:00:32.269057 13161 ts_tablet_manager.cc:1397] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:00:32.270793 13161 raft_consensus.cc:348] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3f8a48ffc44feba530e54fac654770" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 42309 } } peers { permanent_uuid: "3bea16947c3245129943a248f3bcc222" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36943 } } peers { permanent_uuid: "47b24d1dc7e44371bbfe102ed27f2984" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 33993 } }
I20260926 15:00:32.270944 13161 raft_consensus.cc:374] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:32.270989 13161 raft_consensus.cc:729] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e3f8a48ffc44feba530e54fac654770, State: Initialized, Role: FOLLOWER
I20260926 15:00:32.271131 13161 consensus_queue.cc:260] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3f8a48ffc44feba530e54fac654770" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 42309 } } peers { permanent_uuid: "3bea16947c3245129943a248f3bcc222" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36943 } } peers { permanent_uuid: "47b24d1dc7e44371bbfe102ed27f2984" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 33993 } }
I20260926 15:00:32.271426 13161 ts_tablet_manager.cc:1428] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 15:00:32.271438 13007 heartbeater.cc:503] Master 127.8.102.62:37767 was elected leader, sending a full tablet report...
I20260926 15:00:32.293562 13164 raft_consensus.cc:482] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:32.293726 13164 raft_consensus.cc:504] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3f8a48ffc44feba530e54fac654770" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 42309 } } peers { permanent_uuid: "3bea16947c3245129943a248f3bcc222" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36943 } } peers { permanent_uuid: "47b24d1dc7e44371bbfe102ed27f2984" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 33993 } }
I20260926 15:00:32.294108 13164 leader_election.cc:288] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9e3f8a48ffc44feba530e54fac654770 (127.8.102.2:42309), 3bea16947c3245129943a248f3bcc222 (127.8.102.1:36943)
I20260926 15:00:32.295631 13165 raft_consensus.cc:482] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:32.295758 13165 raft_consensus.cc:504] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3f8a48ffc44feba530e54fac654770" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 42309 } } peers { permanent_uuid: "3bea16947c3245129943a248f3bcc222" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36943 } } peers { permanent_uuid: "47b24d1dc7e44371bbfe102ed27f2984" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 33993 } }
I20260926 15:00:32.296274 13165 leader_election.cc:288] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3bea16947c3245129943a248f3bcc222 (127.8.102.1:36943), 47b24d1dc7e44371bbfe102ed27f2984 (127.8.102.3:33993)
I20260926 15:00:32.298478 13166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.294403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:36943 (local address 127.8.102.3:49647)
0926 15:00:32.294607 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:32.294620 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:32.294725 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:32.294856 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:32.294857 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:32.294859 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:32.294925 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:32.294927 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.295794 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.295795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:32.297156 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:32.297159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.298222 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.298223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:32.298295 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:32.298299 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:00:32.298345 (+ 46us) client_negotiation.cc:240] Negotiation successful
0926 15:00:32.298387 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":37,"thread_start_us":122,"threads_started":1}
I20260926 15:00:32.298707 13156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.294503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:49647 (local address 127.8.102.1:36943)
0926 15:00:32.294564 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:32.294566 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:32.294711 (+ 145us) server_negotiation.cc:407] Connection header received
0926 15:00:32.294785 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:32.294786 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:32.294795 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:32.294838 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:32.295002 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.295690 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.297258 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.298099 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.298182 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:32.298192 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:32.298574 (+ 382us) server_negotiation.cc:299] Negotiation successful
0926 15:00:32.298621 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 15:00:32.299463 13008 tablet.cc:2369] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:00:32.300515 12828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b64366e12c479aa8a17b9119e066bd" candidate_uuid: "47b24d1dc7e44371bbfe102ed27f2984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bea16947c3245129943a248f3bcc222" is_pre_election: true
I20260926 15:00:32.300685 12828 raft_consensus.cc:2454] T f9b64366e12c479aa8a17b9119e066bd P 3bea16947c3245129943a248f3bcc222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47b24d1dc7e44371bbfe102ed27f2984 in term 0.
I20260926 15:00:32.300976 13029 leader_election.cc:302] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bea16947c3245129943a248f3bcc222, 47b24d1dc7e44371bbfe102ed27f2984; no voters:
I20260926 15:00:32.301124 13164 raft_consensus.cc:2781] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:00:32.301178 13164 raft_consensus.cc:482] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:00:32.301200 13164 raft_consensus.cc:3037] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:32.301254 13153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.294374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:54515 (local address 127.8.102.2:42309)
0926 15:00:32.294435 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:32.294438 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:32.294449 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:00:32.294595 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:32.294596 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:32.294608 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:32.294710 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:32.294901 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.295690 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.297257 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.297837 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.297904 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:32.297915 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:32.301148 (+ 3233us) server_negotiation.cc:299] Negotiation successful
0926 15:00:32.301196 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 15:00:32.299587 13142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.294287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:42309 (local address 127.8.102.3:54515)
0926 15:00:32.294354 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:32.294370 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:32.294438 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:32.294734 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:32.294735 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:32.294737 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:32.294822 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:32.294824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.295794 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.295795 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:32.297156 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:32.297159 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.297947 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.297947 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:32.298029 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:32.298032 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:00:32.299464 (+ 1432us) client_negotiation.cc:240] Negotiation successful
0926 15:00:32.299510 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":69}
I20260926 15:00:32.301463 13009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.296257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:36943 (local address 127.8.102.2:38531)
0926 15:00:32.296538 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:32.296551 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:32.296600 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:32.297029 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:32.297030 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:32.297032 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:32.297108 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:32.297110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.299002 (+ 1892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.299003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:32.300238 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:32.300242 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.301188 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.301189 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:32.301271 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:32.301275 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:00:32.301308 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 15:00:32.301365 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":261}
I20260926 15:00:32.301618 12961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b64366e12c479aa8a17b9119e066bd" candidate_uuid: "47b24d1dc7e44371bbfe102ed27f2984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e3f8a48ffc44feba530e54fac654770" is_pre_election: true
I20260926 15:00:32.301725 12961 raft_consensus.cc:2454] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47b24d1dc7e44371bbfe102ed27f2984 in term 0.
I20260926 15:00:32.301841 13167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.296415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:38531 (local address 127.8.102.1:36943)
0926 15:00:32.296883 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:32.296885 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:32.296893 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:00:32.296916 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:32.296917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:32.296927 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:32.297178 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:32.297217 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.298892 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.300438 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.301090 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.301666 (+ 576us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:32.301680 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:32.301701 (+ 21us) server_negotiation.cc:299] Negotiation successful
0926 15:00:32.301744 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":52,"server-negotiator.queue_time_us":442,"thread_start_us":113,"threads_started":1}
I20260926 15:00:32.302033 12828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b64366e12c479aa8a17b9119e066bd" candidate_uuid: "9e3f8a48ffc44feba530e54fac654770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bea16947c3245129943a248f3bcc222" is_pre_election: true
I20260926 15:00:32.302124 12828 raft_consensus.cc:2454] T f9b64366e12c479aa8a17b9119e066bd P 3bea16947c3245129943a248f3bcc222 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9e3f8a48ffc44feba530e54fac654770 in term 0.
I20260926 15:00:32.302210 13164 raft_consensus.cc:504] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3f8a48ffc44feba530e54fac654770" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 42309 } } peers { permanent_uuid: "3bea16947c3245129943a248f3bcc222" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36943 } } peers { permanent_uuid: "47b24d1dc7e44371bbfe102ed27f2984" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 33993 } }
I20260926 15:00:32.302289 12896 leader_election.cc:302] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bea16947c3245129943a248f3bcc222, 9e3f8a48ffc44feba530e54fac654770; no voters:
I20260926 15:00:32.302335 13164 leader_election.cc:288] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [CANDIDATE]: Term 1 election: Requested vote from peers 9e3f8a48ffc44feba530e54fac654770 (127.8.102.2:42309), 3bea16947c3245129943a248f3bcc222 (127.8.102.1:36943)
I20260926 15:00:32.302430 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.296449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:43871 (local address 127.8.102.3:33993)
0926 15:00:32.296687 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:32.296689 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:32.296920 (+ 231us) server_negotiation.cc:407] Connection header received
0926 15:00:32.297044 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:32.297045 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:32.297055 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:32.297089 (+ 34us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:32.297307 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.297986 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.300400 (+ 2414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.301006 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.301057 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:32.301066 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:32.302258 (+ 1192us) server_negotiation.cc:299] Negotiation successful
0926 15:00:32.302302 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205}
I20260926 15:00:32.302466 12961 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b64366e12c479aa8a17b9119e066bd" candidate_uuid: "47b24d1dc7e44371bbfe102ed27f2984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9e3f8a48ffc44feba530e54fac654770"
I20260926 15:00:32.302521 12961 raft_consensus.cc:3037] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:32.302534 12828 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b64366e12c479aa8a17b9119e066bd" candidate_uuid: "47b24d1dc7e44371bbfe102ed27f2984" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3bea16947c3245129943a248f3bcc222"
I20260926 15:00:32.302587 12828 raft_consensus.cc:3037] T f9b64366e12c479aa8a17b9119e066bd P 3bea16947c3245129943a248f3bcc222 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:32.302788 13168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.296257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:33993 (local address 127.8.102.2:43871)
0926 15:00:32.296851 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:32.296863 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:32.296949 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:32.297152 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:32.297153 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:32.297155 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:32.297220 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:32.297222 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.298104 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.298105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:32.300296 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:32.300297 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.301812 (+ 1515us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.301813 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:32.302064 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:32.302070 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 15:00:32.302422 (+ 352us) client_negotiation.cc:240] Negotiation successful
0926 15:00:32.302649 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":577,"mutex_wait_us":442,"thread_start_us":100,"threads_started":1}
I20260926 15:00:32.303308 13094 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f9b64366e12c479aa8a17b9119e066bd" candidate_uuid: "9e3f8a48ffc44feba530e54fac654770" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47b24d1dc7e44371bbfe102ed27f2984" is_pre_election: true
I20260926 15:00:32.303390 12961 raft_consensus.cc:2454] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b24d1dc7e44371bbfe102ed27f2984 in term 1.
I20260926 15:00:32.303434 12828 raft_consensus.cc:2454] T f9b64366e12c479aa8a17b9119e066bd P 3bea16947c3245129943a248f3bcc222 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47b24d1dc7e44371bbfe102ed27f2984 in term 1.
I20260926 15:00:32.303439 13094 raft_consensus.cc:2379] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 9e3f8a48ffc44feba530e54fac654770 in current term 1: Already voted for candidate 47b24d1dc7e44371bbfe102ed27f2984 in this term.
I20260926 15:00:32.303546 13165 raft_consensus.cc:2741] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260926 15:00:32.303617 13029 leader_election.cc:302] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3bea16947c3245129943a248f3bcc222, 47b24d1dc7e44371bbfe102ed27f2984; no voters:
I20260926 15:00:32.303731 13164 raft_consensus.cc:2781] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:32.303902 13164 raft_consensus.cc:686] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [term 1 LEADER]: Becoming Leader. State: Replica: 47b24d1dc7e44371bbfe102ed27f2984, State: Running, Role: LEADER
I20260926 15:00:32.304042 13164 consensus_queue.cc:237] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3f8a48ffc44feba530e54fac654770" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 42309 } } peers { permanent_uuid: "3bea16947c3245129943a248f3bcc222" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36943 } } peers { permanent_uuid: "47b24d1dc7e44371bbfe102ed27f2984" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 33993 } }
I20260926 15:00:32.304888 12684 catalog_manager.cc:5495] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 reported cstate change: term changed from 0 to 1, leader changed from <none> to 47b24d1dc7e44371bbfe102ed27f2984 (127.8.102.3). New cstate: current_term: 1 leader_uuid: "47b24d1dc7e44371bbfe102ed27f2984" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e3f8a48ffc44feba530e54fac654770" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 42309 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3bea16947c3245129943a248f3bcc222" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "47b24d1dc7e44371bbfe102ed27f2984" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 33993 } health_report { overall_health: HEALTHY } } }
I20260926 15:00:32.317737 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.315372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51344 (local address 127.8.102.3:33993)
0926 15:00:32.315428 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:32.315430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:32.315440 (+ 10us) server_negotiation.cc:407] Connection header received
0926 15:00:32.315499 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:32.315500 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:32.315517 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:32.315587 (+ 70us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:32.315709 (+ 122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.316184 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.316963 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:32.317152 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:32.317200 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:32.317379 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:32.317478 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:32.317556 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:32.317637 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 15:00:32.317656 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 15:00:32.319645 12828 raft_consensus.cc:1261] T f9b64366e12c479aa8a17b9119e066bd P 3bea16947c3245129943a248f3bcc222 [term 1 FOLLOWER]: Refusing update from remote peer 47b24d1dc7e44371bbfe102ed27f2984: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:32.319677 12961 raft_consensus.cc:1261] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770 [term 1 FOLLOWER]: Refusing update from remote peer 47b24d1dc7e44371bbfe102ed27f2984: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:32.320007 13169 consensus_queue.cc:1035] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3bea16947c3245129943a248f3bcc222" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:32.320154 13164 consensus_queue.cc:1035] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9e3f8a48ffc44feba530e54fac654770" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 42309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:32.322839 13174 mvcc.cc:205] Tried to move back new op lower bound from 7333621073178292224 to 7333621073118130176. Current Snapshot: MvccSnapshot[applied={T|T < 7333621073178292224}]
I20260926 15:00:32.322990 13173 mvcc.cc:205] Tried to move back new op lower bound from 7333621073178292224 to 7333621073118130176. Current Snapshot: MvccSnapshot[applied={T|T < 7333621073178292224}]
I20260926 15:00:32.324824 12684 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:32.324949 12684 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54938:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:32.325785 12684 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 60fb11cfccb341dda185cc07c73dd885: Sending DeleteTablet for 3 replicas of tablet f9b64366e12c479aa8a17b9119e066bd
I20260926 15:00:32.326155 13074 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b64366e12c479aa8a17b9119e066bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:32 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:39222
I20260926 15:00:32.326166 12941 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b64366e12c479aa8a17b9119e066bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:32 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:36216
I20260926 15:00:32.326192 12808 tablet_service.cc:1514] Processing DeleteTablet for tablet f9b64366e12c479aa8a17b9119e066bd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:32 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:36680
I20260926 15:00:32.326716 13182 tablet_replica.cc:330] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984: stopping tablet replica
I20260926 15:00:32.326824 13182 raft_consensus.cc:2229] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:00:32.326872 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 12746
I20260926 15:00:32.327016 13183 tablet_replica.cc:330] T f9b64366e12c479aa8a17b9119e066bd P 3bea16947c3245129943a248f3bcc222: stopping tablet replica
I20260926 15:00:32.327123 13183 raft_consensus.cc:2229] T f9b64366e12c479aa8a17b9119e066bd P 3bea16947c3245129943a248f3bcc222 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:32.328076 13182 raft_consensus.cc:2258] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:32.328615 13182 ts_tablet_manager.cc:1905] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:32.330348 13184 tablet_replica.cc:330] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770: stopping tablet replica
I20260926 15:00:32.330485 13184 raft_consensus.cc:2229] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:32.330520 13182 ts_tablet_manager.cc:1918] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:00:32.330610 13184 raft_consensus.cc:2258] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:32.330732 13182 log.cc:1196] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/f9b64366e12c479aa8a17b9119e066bd
I20260926 15:00:32.331100 13182 ts_tablet_manager.cc:1939] T f9b64366e12c479aa8a17b9119e066bd P 47b24d1dc7e44371bbfe102ed27f2984: Deleting consensus metadata
I20260926 15:00:32.331122 13184 ts_tablet_manager.cc:1905] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:32.331674 12669 catalog_manager.cc:4841] TS 47b24d1dc7e44371bbfe102ed27f2984 (127.8.102.3:33993): tablet f9b64366e12c479aa8a17b9119e066bd (table test-table [id=e90d3ae8bdf84614b2ffba79ddaed866]) successfully deleted
I20260926 15:00:32.333089 13184 ts_tablet_manager.cc:1918] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
W20260926 15:00:32.333122 12670 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.102.1:36943 (error 108)
I20260926 15:00:32.333171 13184 log.cc:1196] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/f9b64366e12c479aa8a17b9119e066bd
I20260926 15:00:32.333487 13184 ts_tablet_manager.cc:1939] T f9b64366e12c479aa8a17b9119e066bd P 9e3f8a48ffc44feba530e54fac654770: Deleting consensus metadata
I20260926 15:00:32.333647 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.333511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:36943 (local address 127.0.0.1:60698)
0926 15:00:32.333566 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:32.333600 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.1:36943: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 15:00:32.333680 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 12879
W20260926 15:00:32.339186 12670 catalog_manager.cc:4584] TS 3bea16947c3245129943a248f3bcc222 (127.8.102.1:36943): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f9b64366e12c479aa8a17b9119e066bd: Network error: Client connection negotiation failed: client connection to 127.8.102.1:36943: connect: Connection refused (error 111)
I20260926 15:00:32.339989 13155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.339855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:42309 (local address 127.0.0.1:42676)
0926 15:00:32.339921 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:32.339956 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.2:42309: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 15:00:32.340112 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 13012
I20260926 15:00:32.344867 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 12653
2026-09-26T15:00:32Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1965 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-26T15:00:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:00:32Z Disabled control of system clock
I20260926 15:00:32.372365 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:34451
--webserver_interface=127.8.102.62
--webserver_port=0
--builtin_ntp_servers=127.8.102.20:39403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:34451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 15:00:32.453586 13189 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:32.453796 13189 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:32.453819 13189 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:32.454909 13189 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:32.454953 13189 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:32.454967 13189 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:32.454978 13189 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:32.454990 13189 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:00:32.456272 13189 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:39403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:34451
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:34451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13189
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:32.456564 13189 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:32.456832 13189 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:32.461905 13197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:32.461871 13195 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:32.461850 13194 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:32.462082 13189 server_base.cc:1034] running on GCE node
I20260926 15:00:32.462332 13189 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:32.462721 13189 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:32.463982 13189 hybrid_clock.cc:648] HybridClock initialized: now 1790434832463955 us; error 44 us; skew 500 ppm
I20260926 15:00:32.465619 13189 webserver.cc:460] Webserver started at http://127.8.102.62:46513/ using document root <none> and password file <none>
I20260926 15:00:32.465894 13189 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:32.465965 13189 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:32.466130 13189 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:32.467344 13189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "26fcea7d9dcc40d7998d2bdfa80947c1"
format_stamp: "Formatted at 2026-09-26 15:00:32 on dist-test-slave-jnhs"
server_key: "901f72ae509a264f11315b76d2fbcd3c"
server_key_iv: "6c5c7027b7caa1d5f8eee45f6de25cfe"
server_key_version: "encryptionkey@0"
I20260926 15:00:32.467712 13189 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "26fcea7d9dcc40d7998d2bdfa80947c1"
format_stamp: "Formatted at 2026-09-26 15:00:32 on dist-test-slave-jnhs"
server_key: "901f72ae509a264f11315b76d2fbcd3c"
server_key_iv: "6c5c7027b7caa1d5f8eee45f6de25cfe"
server_key_version: "encryptionkey@0"
I20260926 15:00:32.469095 13189 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:00:32.469941 13203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:32.470252 13189 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:32.470358 13189 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "26fcea7d9dcc40d7998d2bdfa80947c1"
format_stamp: "Formatted at 2026-09-26 15:00:32 on dist-test-slave-jnhs"
server_key: "901f72ae509a264f11315b76d2fbcd3c"
server_key_iv: "6c5c7027b7caa1d5f8eee45f6de25cfe"
server_key_version: "encryptionkey@0"
I20260926 15:00:32.470436 13189 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:32.474021 13189 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:32.474329 13189 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:32.474473 13189 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:32.478080 13189 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:34451
I20260926 15:00:32.478152 13255 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:34451 every 8 connection(s)
I20260926 15:00:32.478488 13189 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:00:32.479277 13256 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:32.482299 13256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1: Bootstrap starting.
I20260926 15:00:32.483114 13256 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:32.483446 13256 log.cc:824] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:32.484232 13256 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1: No bootstrap required, opened a new log
I20260926 15:00:32.486037 13256 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fcea7d9dcc40d7998d2bdfa80947c1" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 34451 } }
I20260926 15:00:32.486195 13256 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:32.486219 13256 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26fcea7d9dcc40d7998d2bdfa80947c1, State: Initialized, Role: FOLLOWER
I20260926 15:00:32.486409 13256 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fcea7d9dcc40d7998d2bdfa80947c1" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 34451 } }
I20260926 15:00:32.486477 13256 raft_consensus.cc:388] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:00:32.486526 13256 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:00:32.486581 13256 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:32.486980 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 13189
I20260926 15:00:32.487061 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:00:32.487232 8600 external_mini_cluster.cc:1428] Setting key ba3558847ab00c653b1b715cf8d1e716
I20260926 15:00:32.487340 13256 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fcea7d9dcc40d7998d2bdfa80947c1" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 34451 } }
I20260926 15:00:32.487468 13256 leader_election.cc:302] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26fcea7d9dcc40d7998d2bdfa80947c1; no voters:
I20260926 15:00:32.487725 13259 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:32.487870 13259 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1 [term 1 LEADER]: Becoming Leader. State: Replica: 26fcea7d9dcc40d7998d2bdfa80947c1, State: Running, Role: LEADER
I20260926 15:00:32.487671 13256 leader_election.cc:288] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:00:32.487973 13259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fcea7d9dcc40d7998d2bdfa80947c1" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 34451 } }
I20260926 15:00:32.488188 13256 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:32.488317 13259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26fcea7d9dcc40d7998d2bdfa80947c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fcea7d9dcc40d7998d2bdfa80947c1" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 34451 } } }
I20260926 15:00:32.488389 13259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:32.488350 13261 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26fcea7d9dcc40d7998d2bdfa80947c1. Latest consensus state: current_term: 1 leader_uuid: "26fcea7d9dcc40d7998d2bdfa80947c1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26fcea7d9dcc40d7998d2bdfa80947c1" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 34451 } } }
I20260926 15:00:32.488610 13261 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:32.489048 13265 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:00:32.489624 13265 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:00:32.489827 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.487800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60302 (local address 127.8.102.62:34451)
0926 15:00:32.488129 (+ 329us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:32.488138 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:32.488170 (+ 32us) server_negotiation.cc:407] Connection header received
0926 15:00:32.488325 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:32.488333 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:32.488380 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:32.488795 (+ 415us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:32.489346 (+ 551us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:32.489350 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:32.489350 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:00:32.489355 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:32.489393 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:00:32.489407 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:32.489473 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:32.489584 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 15:00:32.489642 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":226,"thread_start_us":80,"threads_started":1}
I20260926 15:00:32.491389 13265 catalog_manager.cc:1347] Generated new cluster ID: d0a6b09b0992464fb9a321f341f217a0
I20260926 15:00:32.491442 13265 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:00:32.503014 13265 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:00:32.503216 13265 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:00:32.518513 13265 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1: Generated new TSK 0
I20260926 15:00:32.518792 13265 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:00:32.523039 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:34451
--builtin_ntp_servers=127.8.102.20:39403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 15:00:32.606618 13280 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:32.606851 13280 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:32.606875 13280 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:32.608111 13280 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:32.608160 13280 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:32.608188 13280 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:00:32.609581 13280 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:39403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:34451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13280
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:32.609853 13280 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:32.610193 13280 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:32.615208 13285 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:32.615271 13286 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:32.615240 13288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:32.615653 13280 server_base.cc:1034] running on GCE node
I20260926 15:00:32.615882 13280 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:32.616156 13280 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:32.617328 13280 hybrid_clock.cc:648] HybridClock initialized: now 1790434832617310 us; error 35 us; skew 500 ppm
I20260926 15:00:32.618957 13280 webserver.cc:460] Webserver started at http://127.8.102.1:39497/ using document root <none> and password file <none>
I20260926 15:00:32.619167 13280 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:32.619210 13280 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:32.619302 13280 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:32.620312 13280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "e70f2cd9b75544e7abc6403f82936e0f"
format_stamp: "Formatted at 2026-09-26 15:00:32 on dist-test-slave-jnhs"
server_key: "2a1dbb9c6dfdaf5fe5caff2ab84f97ca"
server_key_iv: "897b3d8d71f7c22e94b7deccddf96da8"
server_key_version: "encryptionkey@0"
I20260926 15:00:32.620623 13280 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "e70f2cd9b75544e7abc6403f82936e0f"
format_stamp: "Formatted at 2026-09-26 15:00:32 on dist-test-slave-jnhs"
server_key: "2a1dbb9c6dfdaf5fe5caff2ab84f97ca"
server_key_iv: "897b3d8d71f7c22e94b7deccddf96da8"
server_key_version: "encryptionkey@0"
I20260926 15:00:32.621893 13280 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:00:32.622628 13294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:32.622884 13280 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:32.622960 13280 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "e70f2cd9b75544e7abc6403f82936e0f"
format_stamp: "Formatted at 2026-09-26 15:00:32 on dist-test-slave-jnhs"
server_key: "2a1dbb9c6dfdaf5fe5caff2ab84f97ca"
server_key_iv: "897b3d8d71f7c22e94b7deccddf96da8"
server_key_version: "encryptionkey@0"
I20260926 15:00:32.623055 13280 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:32.626359 13280 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:32.626672 13280 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:32.626812 13280 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:32.627034 13280 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:32.627348 13280 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:32.627381 13280 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:32.627401 13280 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:32.627414 13280 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:32.633249 13280 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:43701
I20260926 15:00:32.633297 13407 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:43701 every 8 connection(s)
I20260926 15:00:32.633621 13280 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:00:32.636523 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.634358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:33951 (local address 127.8.102.62:34451)
0926 15:00:32.634425 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:32.634427 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:32.634676 (+ 249us) server_negotiation.cc:407] Connection header received
0926 15:00:32.635150 (+ 474us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:32.635151 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:32.635157 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:32.635210 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:32.636275 (+ 1065us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:32.636276 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:32.636276 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:00:32.636280 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:32.636294 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:00:32.636298 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:32.636358 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:32.636452 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 15:00:32.636467 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 15:00:32.636780 13410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.634347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:34451 (local address 127.8.102.1:33951)
0926 15:00:32.634519 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:32.634552 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:32.635062 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:32.635264 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:32.635268 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:32.635275 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:32.636132 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:00:32.636133 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:32.636162 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:00:32.636191 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:32.636380 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:32.636385 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:32.636387 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 15:00:32.636448 (+ 61us) client_negotiation.cc:240] Negotiation successful
0926 15:00:32.636529 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":86,"threads_started":1}
I20260926 15:00:32.637387 13408 heartbeater.cc:347] Connected to a master server at 127.8.102.62:34451
I20260926 15:00:32.637521 13408 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:32.637681 13408 heartbeater.cc:511] Master 127.8.102.62:34451 requested a full tablet report, sending...
I20260926 15:00:32.638252 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 13280
I20260926 15:00:32.638278 13220 ts_manager.cc:194] Registered new tserver with Master: e70f2cd9b75544e7abc6403f82936e0f (127.8.102.1:43701)
I20260926 15:00:32.638367 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:00:32.638612 8600 external_mini_cluster.cc:1428] Setting key 003791b647d78575cfe0d5009265bde0
I20260926 15:00:32.639616 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:34451
--builtin_ntp_servers=127.8.102.20:39403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260926 15:00:32.723937 13411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:32.724285 13411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:32.724341 13411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:32.725790 13411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:32.725862 13411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:32.725924 13411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:00:32.728148 13411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:39403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:34451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:32.728446 13411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:32.728886 13411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:32.734436 13419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:32.734498 13417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:32.734436 13416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:32.734611 13411 server_base.cc:1034] running on GCE node
I20260926 15:00:32.734882 13411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:32.735114 13411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:32.736266 13411 hybrid_clock.cc:648] HybridClock initialized: now 1790434832736251 us; error 36 us; skew 500 ppm
I20260926 15:00:32.737582 13411 webserver.cc:460] Webserver started at http://127.8.102.2:46851/ using document root <none> and password file <none>
I20260926 15:00:32.737782 13411 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:32.737916 13411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:32.738059 13411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:32.739261 13411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "b85724a6d0b242bf98cf2c27c4380961"
format_stamp: "Formatted at 2026-09-26 15:00:32 on dist-test-slave-jnhs"
server_key: "b084bdce50434f6887d1d1a52118a636"
server_key_iv: "6df8762b46312673a29da306a0129f7a"
server_key_version: "encryptionkey@0"
I20260926 15:00:32.739619 13411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "b85724a6d0b242bf98cf2c27c4380961"
format_stamp: "Formatted at 2026-09-26 15:00:32 on dist-test-slave-jnhs"
server_key: "b084bdce50434f6887d1d1a52118a636"
server_key_iv: "6df8762b46312673a29da306a0129f7a"
server_key_version: "encryptionkey@0"
I20260926 15:00:32.740984 13411 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:00:32.741751 13425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:32.741937 13411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:32.742012 13411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "b85724a6d0b242bf98cf2c27c4380961"
format_stamp: "Formatted at 2026-09-26 15:00:32 on dist-test-slave-jnhs"
server_key: "b084bdce50434f6887d1d1a52118a636"
server_key_iv: "6df8762b46312673a29da306a0129f7a"
server_key_version: "encryptionkey@0"
I20260926 15:00:32.742071 13411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:32.745508 13411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:32.745738 13411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:32.745834 13411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:32.746002 13411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:32.746246 13411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:32.746274 13411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:32.746294 13411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:32.746307 13411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:32.752041 13411 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:36093
I20260926 15:00:32.752122 13538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:36093 every 8 connection(s)
I20260926 15:00:32.752434 13411 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:00:32.754873 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 13411
I20260926 15:00:32.754953 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:00:32.755137 8600 external_mini_cluster.cc:1428] Setting key 9aae97e47a696542adfbfb8f0b328c1c
I20260926 15:00:32.755810 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.753214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:58851 (local address 127.8.102.62:34451)
0926 15:00:32.753279 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:32.753281 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:32.753622 (+ 341us) server_negotiation.cc:407] Connection header received
0926 15:00:32.754128 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:32.754129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:32.754135 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:32.754193 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:32.755562 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:32.755564 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:32.755564 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:00:32.755569 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:32.755589 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:00:32.755594 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:32.755664 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:32.755717 (+ 53us) server_negotiation.cc:299] Negotiation successful
0926 15:00:32.755731 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 15:00:32.756049 13541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.753205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:34451 (local address 127.8.102.2:58851)
0926 15:00:32.753405 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:32.753437 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:32.754012 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:32.754268 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:32.754271 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:32.754279 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:32.755366 (+ 1087us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:00:32.755366 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:32.755397 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:00:32.755450 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:32.755660 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:32.755664 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:32.755666 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 15:00:32.755722 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 15:00:32.755792 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":88,"threads_started":1}
I20260926 15:00:32.756318 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:34451
--builtin_ntp_servers=127.8.102.20:39403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260926 15:00:32.756588 13539 heartbeater.cc:347] Connected to a master server at 127.8.102.62:34451
I20260926 15:00:32.756680 13539 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:32.756830 13539 heartbeater.cc:511] Master 127.8.102.62:34451 requested a full tablet report, sending...
I20260926 15:00:32.757185 13220 ts_manager.cc:194] Registered new tserver with Master: b85724a6d0b242bf98cf2c27c4380961 (127.8.102.2:36093)
W20260926 15:00:32.841010 13542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:32.841259 13542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:32.841281 13542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:32.842425 13542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:32.842464 13542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:32.842489 13542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:00:32.843888 13542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:39403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:34451
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:32.844118 13542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:32.844377 13542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:32.849454 13550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:32.849763 13542 server_base.cc:1034] running on GCE node
W20260926 15:00:32.849900 13547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:32.849900 13548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:32.850236 13542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:32.850508 13542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:32.851697 13542 hybrid_clock.cc:648] HybridClock initialized: now 1790434832851692 us; error 57 us; skew 500 ppm
I20260926 15:00:32.853012 13542 webserver.cc:460] Webserver started at http://127.8.102.3:39447/ using document root <none> and password file <none>
I20260926 15:00:32.853268 13542 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:32.853338 13542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:32.853535 13542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:32.854698 13542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "0f1e8dca01994086972c7379ed316857"
format_stamp: "Formatted at 2026-09-26 15:00:32 on dist-test-slave-jnhs"
server_key: "b90790c46fb2e121cc49755c8664cea2"
server_key_iv: "65437aea7ab2b89a147a499eb2566778"
server_key_version: "encryptionkey@0"
I20260926 15:00:32.855104 13542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "0f1e8dca01994086972c7379ed316857"
format_stamp: "Formatted at 2026-09-26 15:00:32 on dist-test-slave-jnhs"
server_key: "b90790c46fb2e121cc49755c8664cea2"
server_key_iv: "65437aea7ab2b89a147a499eb2566778"
server_key_version: "encryptionkey@0"
I20260926 15:00:32.856594 13542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 15:00:32.857501 13556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:32.857698 13542 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:32.857805 13542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "0f1e8dca01994086972c7379ed316857"
format_stamp: "Formatted at 2026-09-26 15:00:32 on dist-test-slave-jnhs"
server_key: "b90790c46fb2e121cc49755c8664cea2"
server_key_iv: "65437aea7ab2b89a147a499eb2566778"
server_key_version: "encryptionkey@0"
I20260926 15:00:32.857894 13542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:32.861404 13542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:32.861696 13542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:32.861835 13542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:32.862046 13542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:32.862385 13542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:32.862438 13542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:32.862505 13542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:32.862536 13542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:32.868404 13542 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:45977
I20260926 15:00:32.868479 13669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:45977 every 8 connection(s)
I20260926 15:00:32.868824 13542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:00:32.871577 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 13542
I20260926 15:00:32.871716 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:00:32.871938 8600 external_mini_cluster.cc:1428] Setting key 932dbaee4598cb0be6635f76ac4ee488
I20260926 15:00:32.872102 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.869642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:52465 (local address 127.8.102.62:34451)
0926 15:00:32.869699 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:32.869701 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:32.869921 (+ 220us) server_negotiation.cc:407] Connection header received
0926 15:00:32.870365 (+ 444us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:32.870366 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:32.870370 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:32.870423 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:32.871746 (+ 1323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:32.871748 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:32.871749 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:00:32.871754 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:32.871772 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:00:32.871777 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:32.871844 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:32.871995 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 15:00:32.872016 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 15:00:32.872351 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.869633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:34451 (local address 127.8.102.3:52465)
0926 15:00:32.869798 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:32.869832 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:32.870282 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:32.870478 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:32.870481 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:32.870489 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:32.871490 (+ 1001us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:00:32.871491 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:32.871520 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:00:32.871551 (+ 31us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:32.871876 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:32.871882 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:32.871886 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:00:32.871970 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 15:00:32.872043 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":80,"threads_started":1}
I20260926 15:00:32.872802 13670 heartbeater.cc:347] Connected to a master server at 127.8.102.62:34451
I20260926 15:00:32.872922 13670 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:32.873075 13670 heartbeater.cc:511] Master 127.8.102.62:34451 requested a full tablet report, sending...
I20260926 15:00:32.873438 13220 ts_manager.cc:194] Registered new tserver with Master: 0f1e8dca01994086972c7379ed316857 (127.8.102.3:45977)
I20260926 15:00:32.883962 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:00:32.886749 13262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.885595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60306 (local address 127.8.102.62:34451)
0926 15:00:32.885695 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:32.885697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:32.885782 (+ 85us) server_negotiation.cc:407] Connection header received
0926 15:00:32.885853 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:32.885854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:32.885859 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:32.885896 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:32.886454 (+ 558us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:32.886455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:32.886455 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:00:32.886459 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:32.886482 (+ 23us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:00:32.886490 (+ 8us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:32.886543 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:32.886587 (+ 44us) server_negotiation.cc:299] Negotiation successful
0926 15:00:32.886601 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 15:00:32.888458 13220 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:60306:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:00:32.890352 13220 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:00:32.894459 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.892781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44238 (local address 127.8.102.2:36093)
0926 15:00:32.892957 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:32.892962 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:32.893174 (+ 212us) server_negotiation.cc:407] Connection header received
0926 15:00:32.893207 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:32.893210 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:32.893222 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:32.893304 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:32.894137 (+ 833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:32.894141 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:32.894141 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:00:32.894144 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:32.894172 (+ 28us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:00:32.894193 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:32.894271 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:32.894352 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 15:00:32.894368 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":97,"threads_started":1}
I20260926 15:00:32.894449 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.892671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:36093 (local address 127.0.0.1:44238)
0926 15:00:32.892994 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:32.893038 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:32.893106 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:32.893321 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:32.893325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:32.893331 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:32.893843 (+ 512us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:00:32.893843 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:32.893857 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:00:32.893875 (+ 18us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:32.894279 (+ 404us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:32.894280 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:32.894280 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 15:00:32.894335 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 15:00:32.894354 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":290,"mutex_wait_us":2,"thread_start_us":70,"threads_started":1}
I20260926 15:00:32.894464 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.892879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:43701 (local address 127.0.0.1:60832)
0926 15:00:32.893069 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:32.893078 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:32.893111 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:32.893645 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:32.893646 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:32.893649 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:32.894019 (+ 370us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:00:32.894020 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:32.894027 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:00:32.894041 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:32.894252 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:32.894258 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:32.894261 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:00:32.894333 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 15:00:32.894351 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":57,"threads_started":1}
I20260926 15:00:32.894739 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.893066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60832 (local address 127.8.102.1:43701)
0926 15:00:32.893249 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:32.893253 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:32.893262 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:32.893288 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:32.893290 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:32.893298 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:32.893363 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:32.894126 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:32.894133 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:32.894134 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:00:32.894138 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:32.894167 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:00:32.894186 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:32.894234 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:32.894605 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 15:00:32.894622 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":70,"threads_started":1}
I20260926 15:00:32.895377 13473 tablet_service.cc:1467] Processing CreateTablet for tablet 84dde1e53dd949288d51f401cd24d40e (DEFAULT_TABLE table=test-table [id=1664e4e6a6964c62bfc85346aa58b6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:32.895643 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.892851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52496 (local address 127.8.102.3:45977)
0926 15:00:32.894358 (+ 1507us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:32.894363 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:32.894371 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:00:32.894397 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:32.894399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:32.894409 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:32.894486 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:32.895355 (+ 869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:32.895362 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:32.895362 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:00:32.895366 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:32.895400 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:00:32.895424 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:32.895482 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:32.895550 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 15:00:32.895564 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1467,"thread_start_us":81,"threads_started":1}
I20260926 15:00:32.895834 13473 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84dde1e53dd949288d51f401cd24d40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:32.895853 13679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:32.892671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:45977 (local address 127.0.0.1:52496)
0926 15:00:32.893024 (+ 353us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:32.893039 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:32.893108 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:32.894748 (+ 1640us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:32.894749 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:32.894751 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:32.895257 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:00:32.895257 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:32.895264 (+ 7us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:00:32.895278 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:32.895481 (+ 203us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:32.895482 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:32.895484 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 15:00:32.895542 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 15:00:32.895555 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":334,"mutex_wait_us":2,"thread_start_us":73,"threads_started":1}
I20260926 15:00:32.896502 13604 tablet_service.cc:1467] Processing CreateTablet for tablet 84dde1e53dd949288d51f401cd24d40e (DEFAULT_TABLE table=test-table [id=1664e4e6a6964c62bfc85346aa58b6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:32.896826 13604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84dde1e53dd949288d51f401cd24d40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:32.899271 13342 tablet_service.cc:1467] Processing CreateTablet for tablet 84dde1e53dd949288d51f401cd24d40e (DEFAULT_TABLE table=test-table [id=1664e4e6a6964c62bfc85346aa58b6bb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:32.899570 13342 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84dde1e53dd949288d51f401cd24d40e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:32.899746 13685 tablet_bootstrap.cc:492] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857: Bootstrap starting.
I20260926 15:00:32.900404 13685 tablet_bootstrap.cc:654] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:32.900740 13685 log.cc:824] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:32.901540 13685 tablet_bootstrap.cc:492] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857: No bootstrap required, opened a new log
I20260926 15:00:32.901633 13685 ts_tablet_manager.cc:1397] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:00:32.902925 13684 tablet_bootstrap.cc:492] T 84dde1e53dd949288d51f401cd24d40e P b85724a6d0b242bf98cf2c27c4380961: Bootstrap starting.
I20260926 15:00:32.903565 13684 tablet_bootstrap.cc:654] T 84dde1e53dd949288d51f401cd24d40e P b85724a6d0b242bf98cf2c27c4380961: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:32.903847 13684 log.cc:824] T 84dde1e53dd949288d51f401cd24d40e P b85724a6d0b242bf98cf2c27c4380961: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:32.904139 13685 raft_consensus.cc:348] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e8dca01994086972c7379ed316857" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 45977 } } peers { permanent_uuid: "b85724a6d0b242bf98cf2c27c4380961" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36093 } } peers { permanent_uuid: "e70f2cd9b75544e7abc6403f82936e0f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43701 } }
I20260926 15:00:32.904328 13685 raft_consensus.cc:374] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:32.904405 13685 raft_consensus.cc:729] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f1e8dca01994086972c7379ed316857, State: Initialized, Role: FOLLOWER
I20260926 15:00:32.904551 13685 consensus_queue.cc:260] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e8dca01994086972c7379ed316857" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 45977 } } peers { permanent_uuid: "b85724a6d0b242bf98cf2c27c4380961" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36093 } } peers { permanent_uuid: "e70f2cd9b75544e7abc6403f82936e0f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43701 } }
I20260926 15:00:32.904803 13684 tablet_bootstrap.cc:492] T 84dde1e53dd949288d51f401cd24d40e P b85724a6d0b242bf98cf2c27c4380961: No bootstrap required, opened a new log
I20260926 15:00:32.904863 13685 ts_tablet_manager.cc:1428] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 15:00:32.904898 13670 heartbeater.cc:503] Master 127.8.102.62:34451 was elected leader, sending a full tablet report...
I20260926 15:00:32.904961 13684 ts_tablet_manager.cc:1397] T 84dde1e53dd949288d51f401cd24d40e P b85724a6d0b242bf98cf2c27c4380961: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:00:32.905071 13687 tablet_bootstrap.cc:492] T 84dde1e53dd949288d51f401cd24d40e P e70f2cd9b75544e7abc6403f82936e0f: Bootstrap starting.
I20260926 15:00:32.905715 13687 tablet_bootstrap.cc:654] T 84dde1e53dd949288d51f401cd24d40e P e70f2cd9b75544e7abc6403f82936e0f: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:32.906157 13687 log.cc:824] T 84dde1e53dd949288d51f401cd24d40e P e70f2cd9b75544e7abc6403f82936e0f: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:32.906889 13687 tablet_bootstrap.cc:492] T 84dde1e53dd949288d51f401cd24d40e P e70f2cd9b75544e7abc6403f82936e0f: No bootstrap required, opened a new log
I20260926 15:00:32.906944 13687 ts_tablet_manager.cc:1397] T 84dde1e53dd949288d51f401cd24d40e P e70f2cd9b75544e7abc6403f82936e0f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:00:32.907787 13684 raft_consensus.cc:348] T 84dde1e53dd949288d51f401cd24d40e P b85724a6d0b242bf98cf2c27c4380961 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e8dca01994086972c7379ed316857" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 45977 } } peers { permanent_uuid: "b85724a6d0b242bf98cf2c27c4380961" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36093 } } peers { permanent_uuid: "e70f2cd9b75544e7abc6403f82936e0f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43701 } }
I20260926 15:00:32.907959 13684 raft_consensus.cc:374] T 84dde1e53dd949288d51f401cd24d40e P b85724a6d0b242bf98cf2c27c4380961 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:32.908010 13684 raft_consensus.cc:729] T 84dde1e53dd949288d51f401cd24d40e P b85724a6d0b242bf98cf2c27c4380961 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b85724a6d0b242bf98cf2c27c4380961, State: Initialized, Role: FOLLOWER
I20260926 15:00:32.908143 13684 consensus_queue.cc:260] T 84dde1e53dd949288d51f401cd24d40e P b85724a6d0b242bf98cf2c27c4380961 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e8dca01994086972c7379ed316857" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 45977 } } peers { permanent_uuid: "b85724a6d0b242bf98cf2c27c4380961" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36093 } } peers { permanent_uuid: "e70f2cd9b75544e7abc6403f82936e0f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43701 } }
I20260926 15:00:32.908413 13684 ts_tablet_manager.cc:1428] T 84dde1e53dd949288d51f401cd24d40e P b85724a6d0b242bf98cf2c27c4380961: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 15:00:32.908480 13539 heartbeater.cc:503] Master 127.8.102.62:34451 was elected leader, sending a full tablet report...
I20260926 15:00:32.909089 13687 raft_consensus.cc:348] T 84dde1e53dd949288d51f401cd24d40e P e70f2cd9b75544e7abc6403f82936e0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e8dca01994086972c7379ed316857" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 45977 } } peers { permanent_uuid: "b85724a6d0b242bf98cf2c27c4380961" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36093 } } peers { permanent_uuid: "e70f2cd9b75544e7abc6403f82936e0f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43701 } }
I20260926 15:00:32.909286 13687 raft_consensus.cc:374] T 84dde1e53dd949288d51f401cd24d40e P e70f2cd9b75544e7abc6403f82936e0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:32.909330 13687 raft_consensus.cc:729] T 84dde1e53dd949288d51f401cd24d40e P e70f2cd9b75544e7abc6403f82936e0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e70f2cd9b75544e7abc6403f82936e0f, State: Initialized, Role: FOLLOWER
I20260926 15:00:32.909456 13687 consensus_queue.cc:260] T 84dde1e53dd949288d51f401cd24d40e P e70f2cd9b75544e7abc6403f82936e0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e8dca01994086972c7379ed316857" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 45977 } } peers { permanent_uuid: "b85724a6d0b242bf98cf2c27c4380961" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36093 } } peers { permanent_uuid: "e70f2cd9b75544e7abc6403f82936e0f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43701 } }
I20260926 15:00:32.909737 13687 ts_tablet_manager.cc:1428] T 84dde1e53dd949288d51f401cd24d40e P e70f2cd9b75544e7abc6403f82936e0f: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 15:00:32.909776 13408 heartbeater.cc:503] Master 127.8.102.62:34451 was elected leader, sending a full tablet report...
W20260926 15:00:33.003072 13540 tablet.cc:2369] T 84dde1e53dd949288d51f401cd24d40e P b85724a6d0b242bf98cf2c27c4380961: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:00:33.012146 13689 raft_consensus.cc:482] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:33.012255 13689 raft_consensus.cc:504] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e8dca01994086972c7379ed316857" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 45977 } } peers { permanent_uuid: "b85724a6d0b242bf98cf2c27c4380961" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36093 } } peers { permanent_uuid: "e70f2cd9b75544e7abc6403f82936e0f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43701 } }
I20260926 15:00:33.012562 13689 leader_election.cc:288] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b85724a6d0b242bf98cf2c27c4380961 (127.8.102.2:36093), e70f2cd9b75544e7abc6403f82936e0f (127.8.102.1:43701)
I20260926 15:00:33.013983 13672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:33.012775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:36093 (local address 127.8.102.3:58193)
0926 15:00:33.012839 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:33.012853 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:33.012925 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:33.013089 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:33.013090 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:33.013092 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:33.013572 (+ 480us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:00:33.013573 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:33.013581 (+ 8us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:00:33.013595 (+ 14us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:33.013816 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:33.013817 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:33.013817 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 15:00:33.013893 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 15:00:33.013907 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 15:00:33.014020 13680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:33.012879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:58193 (local address 127.8.102.2:36093)
0926 15:00:33.012931 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:33.012933 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:33.012968 (+ 35us) server_negotiation.cc:407] Connection header received
0926 15:00:33.012996 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:33.012997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:33.013004 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:33.013077 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:33.013693 (+ 616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:33.013695 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:33.013695 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:00:33.013699 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:33.013740 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:00:33.013746 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:33.013813 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:33.013915 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 15:00:33.013928 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 15:00:33.014181 13683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:33.013000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:58281 (local address 127.8.102.1:43701)
0926 15:00:33.013048 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:33.013050 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:33.013099 (+ 49us) server_negotiation.cc:407] Connection header received
0926 15:00:33.013156 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:33.013157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:33.013162 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:33.013243 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:33.013873 (+ 630us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:33.013875 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:33.013876 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:00:33.013881 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:33.013899 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:00:33.013904 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:33.013991 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:33.014094 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 15:00:33.014111 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 15:00:33.014321 13493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84dde1e53dd949288d51f401cd24d40e" candidate_uuid: "0f1e8dca01994086972c7379ed316857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b85724a6d0b242bf98cf2c27c4380961" is_pre_election: true
I20260926 15:00:33.014396 13693 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:33.012890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:43701 (local address 127.8.102.3:58281)
0926 15:00:33.013047 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:33.013056 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:33.013100 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:33.013249 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:33.013250 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:33.013252 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:33.013770 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:00:33.013771 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:33.013777 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:00:33.013789 (+ 12us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:33.014003 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:33.014004 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:33.014004 (+ 0us) client_negotiation.cc:769] Sending connection context
0926 15:00:33.014284 (+ 280us) client_negotiation.cc:240] Negotiation successful
0926 15:00:33.014300 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":109,"threads_started":1}
I20260926 15:00:33.014542 13493 raft_consensus.cc:2454] T 84dde1e53dd949288d51f401cd24d40e P b85724a6d0b242bf98cf2c27c4380961 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f1e8dca01994086972c7379ed316857 in term 0.
I20260926 15:00:33.014819 13557 leader_election.cc:302] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f1e8dca01994086972c7379ed316857, b85724a6d0b242bf98cf2c27c4380961; no voters:
I20260926 15:00:33.014958 13689 raft_consensus.cc:2781] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:00:33.014925 13362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84dde1e53dd949288d51f401cd24d40e" candidate_uuid: "0f1e8dca01994086972c7379ed316857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e70f2cd9b75544e7abc6403f82936e0f" is_pre_election: true
I20260926 15:00:33.015020 13689 raft_consensus.cc:482] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:00:33.015064 13689 raft_consensus.cc:3037] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:33.015081 13362 raft_consensus.cc:2454] T 84dde1e53dd949288d51f401cd24d40e P e70f2cd9b75544e7abc6403f82936e0f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0f1e8dca01994086972c7379ed316857 in term 0.
I20260926 15:00:33.015849 13689 raft_consensus.cc:504] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e8dca01994086972c7379ed316857" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 45977 } } peers { permanent_uuid: "b85724a6d0b242bf98cf2c27c4380961" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36093 } } peers { permanent_uuid: "e70f2cd9b75544e7abc6403f82936e0f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43701 } }
I20260926 15:00:33.015988 13689 leader_election.cc:288] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [CANDIDATE]: Term 1 election: Requested vote from peers b85724a6d0b242bf98cf2c27c4380961 (127.8.102.2:36093), e70f2cd9b75544e7abc6403f82936e0f (127.8.102.1:43701)
I20260926 15:00:33.016161 13362 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84dde1e53dd949288d51f401cd24d40e" candidate_uuid: "0f1e8dca01994086972c7379ed316857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e70f2cd9b75544e7abc6403f82936e0f"
I20260926 15:00:33.016173 13493 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84dde1e53dd949288d51f401cd24d40e" candidate_uuid: "0f1e8dca01994086972c7379ed316857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b85724a6d0b242bf98cf2c27c4380961"
I20260926 15:00:33.016244 13493 raft_consensus.cc:3037] T 84dde1e53dd949288d51f401cd24d40e P b85724a6d0b242bf98cf2c27c4380961 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:33.016264 13362 raft_consensus.cc:3037] T 84dde1e53dd949288d51f401cd24d40e P e70f2cd9b75544e7abc6403f82936e0f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:33.016961 13493 raft_consensus.cc:2454] T 84dde1e53dd949288d51f401cd24d40e P b85724a6d0b242bf98cf2c27c4380961 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f1e8dca01994086972c7379ed316857 in term 1.
I20260926 15:00:33.017103 13362 raft_consensus.cc:2454] T 84dde1e53dd949288d51f401cd24d40e P e70f2cd9b75544e7abc6403f82936e0f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0f1e8dca01994086972c7379ed316857 in term 1.
I20260926 15:00:33.017122 13557 leader_election.cc:302] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f1e8dca01994086972c7379ed316857, b85724a6d0b242bf98cf2c27c4380961; no voters:
I20260926 15:00:33.017336 13689 raft_consensus.cc:2781] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:33.017516 13689 raft_consensus.cc:686] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [term 1 LEADER]: Becoming Leader. State: Replica: 0f1e8dca01994086972c7379ed316857, State: Running, Role: LEADER
I20260926 15:00:33.017619 13689 consensus_queue.cc:237] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e8dca01994086972c7379ed316857" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 45977 } } peers { permanent_uuid: "b85724a6d0b242bf98cf2c27c4380961" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36093 } } peers { permanent_uuid: "e70f2cd9b75544e7abc6403f82936e0f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43701 } }
I20260926 15:00:33.018457 13219 catalog_manager.cc:5495] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0f1e8dca01994086972c7379ed316857 (127.8.102.3). New cstate: current_term: 1 leader_uuid: "0f1e8dca01994086972c7379ed316857" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0f1e8dca01994086972c7379ed316857" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 45977 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b85724a6d0b242bf98cf2c27c4380961" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36093 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e70f2cd9b75544e7abc6403f82936e0f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43701 } health_report { overall_health: UNKNOWN } } }
I20260926 15:00:33.043803 13681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:33.042690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52506 (local address 127.8.102.3:45977)
0926 15:00:33.042747 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:33.042750 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:33.042758 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:00:33.042814 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:33.042815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:33.042828 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:33.042879 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:33.043530 (+ 651us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:33.043531 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:33.043531 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:00:33.043534 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:33.043545 (+ 11us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:00:33.043548 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:33.043593 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:33.043668 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 15:00:33.043680 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 15:00:33.045773 13362 raft_consensus.cc:1261] T 84dde1e53dd949288d51f401cd24d40e P e70f2cd9b75544e7abc6403f82936e0f [term 1 FOLLOWER]: Refusing update from remote peer 0f1e8dca01994086972c7379ed316857: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:33.045822 13493 raft_consensus.cc:1261] T 84dde1e53dd949288d51f401cd24d40e P b85724a6d0b242bf98cf2c27c4380961 [term 1 FOLLOWER]: Refusing update from remote peer 0f1e8dca01994086972c7379ed316857: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:33.046213 13694 consensus_queue.cc:1035] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b85724a6d0b242bf98cf2c27c4380961" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36093 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:33.046357 13689 consensus_queue.cc:1035] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e70f2cd9b75544e7abc6403f82936e0f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:33.048951 13699 mvcc.cc:205] Tried to move back new op lower bound from 7333621076152573952 to 7333621076040871936. Current Snapshot: MvccSnapshot[applied={T|T < 7333621076152573952}]
I20260926 15:00:33.048965 13698 mvcc.cc:205] Tried to move back new op lower bound from 7333621076152573952 to 7333621076040871936. Current Snapshot: MvccSnapshot[applied={T|T < 7333621076152573952}]
I20260926 15:00:33.051949 13219 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:60306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:33.052060 13219 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:60306:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:33.052894 13219 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26fcea7d9dcc40d7998d2bdfa80947c1: Sending DeleteTablet for 3 replicas of tablet 84dde1e53dd949288d51f401cd24d40e
I20260926 15:00:33.053251 13604 tablet_service.cc:1514] Processing DeleteTablet for tablet 84dde1e53dd949288d51f401cd24d40e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:33 UTC) from {username='slave'} at 127.0.0.1:52496
I20260926 15:00:33.053300 13342 tablet_service.cc:1514] Processing DeleteTablet for tablet 84dde1e53dd949288d51f401cd24d40e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:33 UTC) from {username='slave'} at 127.0.0.1:60832
I20260926 15:00:33.053360 13473 tablet_service.cc:1514] Processing DeleteTablet for tablet 84dde1e53dd949288d51f401cd24d40e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:33 UTC) from {username='slave'} at 127.0.0.1:44238
I20260926 15:00:33.053632 13708 tablet_replica.cc:330] T 84dde1e53dd949288d51f401cd24d40e P e70f2cd9b75544e7abc6403f82936e0f: stopping tablet replica
I20260926 15:00:33.053711 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 13280
I20260926 15:00:33.053762 13708 raft_consensus.cc:2229] T 84dde1e53dd949288d51f401cd24d40e P e70f2cd9b75544e7abc6403f82936e0f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:33.053880 13708 raft_consensus.cc:2258] T 84dde1e53dd949288d51f401cd24d40e P e70f2cd9b75544e7abc6403f82936e0f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:33.055598 13709 tablet_replica.cc:330] T 84dde1e53dd949288d51f401cd24d40e P b85724a6d0b242bf98cf2c27c4380961: stopping tablet replica
I20260926 15:00:33.055732 13709 raft_consensus.cc:2229] T 84dde1e53dd949288d51f401cd24d40e P b85724a6d0b242bf98cf2c27c4380961 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:33.055838 13709 raft_consensus.cc:2258] T 84dde1e53dd949288d51f401cd24d40e P b85724a6d0b242bf98cf2c27c4380961 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:33.055951 13707 tablet_replica.cc:330] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857: stopping tablet replica
I20260926 15:00:33.056046 13707 raft_consensus.cc:2229] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:00:33.056176 13707 raft_consensus.cc:2258] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:33.056746 13709 ts_tablet_manager.cc:1905] T 84dde1e53dd949288d51f401cd24d40e P b85724a6d0b242bf98cf2c27c4380961: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:33.058122 13707 ts_tablet_manager.cc:1905] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:33.059677 13707 ts_tablet_manager.cc:1918] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:00:33.059767 13707 log.cc:1196] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/84dde1e53dd949288d51f401cd24d40e
I20260926 15:00:33.060070 13707 ts_tablet_manager.cc:1939] T 84dde1e53dd949288d51f401cd24d40e P 0f1e8dca01994086972c7379ed316857: Deleting consensus metadata
W20260926 15:00:33.060081 13204 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.102.1:43701 (error 108)
I20260926 15:00:33.060299 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 13411
I20260926 15:00:33.060849 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:33.060738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:43701 (local address 127.0.0.1:60848)
0926 15:00:33.060783 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:33.060815 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.1:43701: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260926 15:00:33.061079 13204 catalog_manager.cc:4584] TS e70f2cd9b75544e7abc6403f82936e0f (127.8.102.1:43701): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 84dde1e53dd949288d51f401cd24d40e: Network error: Client connection negotiation failed: client connection to 127.8.102.1:43701: connect: Connection refused (error 111)
I20260926 15:00:33.064719 13207 catalog_manager.cc:4841] TS 0f1e8dca01994086972c7379ed316857 (127.8.102.3:45977): tablet 84dde1e53dd949288d51f401cd24d40e (table test-table [id=1664e4e6a6964c62bfc85346aa58b6bb]) successfully deleted
I20260926 15:00:33.069423 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 13542
I20260926 15:00:33.069541 13682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:33.069416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:36093 (local address 127.0.0.1:44240)
0926 15:00:33.069473 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:33.069502 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.2:36093: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 15:00:33.073675 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 13189
2026-09-26T15:00:33Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (728 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260926 15:00:33.094123 8600 mini_oidc.cc:152] Starting JWKS server
I20260926 15:00:33.095341 8600 webserver.cc:460] Webserver started at https://127.0.0.1:34635/ using document root <none> and password file <none>
I20260926 15:00:33.095428 8600 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 15:00:33.095604 8600 webserver.cc:460] Webserver started at http://127.0.0.1:34327/ using document root <none> and password file <none>
2026-09-26T15:00:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:00:33Z Disabled control of system clock
I20260926 15:00:33.103768 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:38213
--webserver_interface=127.8.102.62
--webserver_port=0
--builtin_ntp_servers=127.8.102.20:46273
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:38213
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34635/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434810362722-8600-0/testchainca.cert with env {}
W20260926 15:00:33.191291 13716 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:33.191629 13716 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:33.191676 13716 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:33.192950 13716 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:33.193038 13716 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:33.193059 13716 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:33.193079 13716 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:33.193106 13716 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 15:00:33.193137 13716 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 15:00:33.193159 13716 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34635/jwks.json
I20260926 15:00:33.194845 13716 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46273
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:38213
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:38213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34635/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434810362722-8600-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13716
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:33.195185 13716 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:33.195425 13716 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:33.200588 13724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:33.200609 13722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:33.200735 13716 server_base.cc:1034] running on GCE node
W20260926 15:00:33.200590 13721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:33.201227 13716 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:33.201551 13716 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:33.202703 13716 hybrid_clock.cc:648] HybridClock initialized: now 1790434833202697 us; error 38 us; skew 500 ppm
I20260926 15:00:33.204566 13716 webserver.cc:460] Webserver started at http://127.8.102.62:42829/ using document root <none> and password file <none>
I20260926 15:00:33.204845 13716 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:33.204939 13716 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:33.205099 13716 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:33.206213 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "8ba957e47869460b838fc1ec9f4be9e8"
format_stamp: "Formatted at 2026-09-26 15:00:33 on dist-test-slave-jnhs"
server_key: "4b41a203aba4b4e139dcc7fead8d447b"
server_key_iv: "e8b50023df400b5bd8d2af695c58377b"
server_key_version: "encryptionkey@0"
I20260926 15:00:33.206700 13716 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "8ba957e47869460b838fc1ec9f4be9e8"
format_stamp: "Formatted at 2026-09-26 15:00:33 on dist-test-slave-jnhs"
server_key: "4b41a203aba4b4e139dcc7fead8d447b"
server_key_iv: "e8b50023df400b5bd8d2af695c58377b"
server_key_version: "encryptionkey@0"
I20260926 15:00:33.208138 13716 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:33.209079 13730 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:33.209316 13716 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:33.209436 13716 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "8ba957e47869460b838fc1ec9f4be9e8"
format_stamp: "Formatted at 2026-09-26 15:00:33 on dist-test-slave-jnhs"
server_key: "4b41a203aba4b4e139dcc7fead8d447b"
server_key_iv: "e8b50023df400b5bd8d2af695c58377b"
server_key_version: "encryptionkey@0"
I20260926 15:00:33.209543 13716 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:33.226861 13716 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:33.227205 13716 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:33.227355 13716 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:33.231393 13716 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:38213
I20260926 15:00:33.231429 13782 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:38213 every 8 connection(s)
I20260926 15:00:33.231817 13716 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:00:33.232832 13783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:33.235286 13783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8: Bootstrap starting.
I20260926 15:00:33.235903 13783 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:33.236215 13783 log.cc:824] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:33.236995 13783 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8: No bootstrap required, opened a new log
I20260926 15:00:33.238617 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 13716
I20260926 15:00:33.238744 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:00:33.238937 8600 external_mini_cluster.cc:1428] Setting key 616b8829818e9ecb13f6edd487a76e51
I20260926 15:00:33.238942 13783 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ba957e47869460b838fc1ec9f4be9e8" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38213 } }
I20260926 15:00:33.239151 13783 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:33.239184 13783 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ba957e47869460b838fc1ec9f4be9e8, State: Initialized, Role: FOLLOWER
I20260926 15:00:33.239300 13783 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ba957e47869460b838fc1ec9f4be9e8" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38213 } }
I20260926 15:00:33.239393 13783 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:00:33.239436 13783 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:00:33.239490 13783 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:33.240231 13783 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ba957e47869460b838fc1ec9f4be9e8" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38213 } }
I20260926 15:00:33.240370 13783 leader_election.cc:302] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ba957e47869460b838fc1ec9f4be9e8; no voters:
I20260926 15:00:33.240533 13783 leader_election.cc:288] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:00:33.240581 13788 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:33.240737 13788 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8 [term 1 LEADER]: Becoming Leader. State: Replica: 8ba957e47869460b838fc1ec9f4be9e8, State: Running, Role: LEADER
I20260926 15:00:33.240828 13783 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:33.240831 13788 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ba957e47869460b838fc1ec9f4be9e8" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38213 } }
I20260926 15:00:33.241204 13788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ba957e47869460b838fc1ec9f4be9e8. Latest consensus state: current_term: 1 leader_uuid: "8ba957e47869460b838fc1ec9f4be9e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ba957e47869460b838fc1ec9f4be9e8" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38213 } } }
I20260926 15:00:33.241294 13788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:33.241210 13790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ba957e47869460b838fc1ec9f4be9e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ba957e47869460b838fc1ec9f4be9e8" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38213 } } }
I20260926 15:00:33.241387 13790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:33.241511 13795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:00:33.242010 13795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:00:33.243782 13795 catalog_manager.cc:1347] Generated new cluster ID: 72d0f33f92b340e3a499f8241f1806c4
I20260926 15:00:33.243837 13795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:00:33.245486 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:33.239498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53548 (local address 127.8.102.62:38213)
0926 15:00:33.239734 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:33.239744 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:33.239779 (+ 35us) server_negotiation.cc:407] Connection header received
0926 15:00:33.239903 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:33.239910 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:33.239999 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:33.240160 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:33.240381 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.242264 (+ 1883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.243302 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.243916 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.244039 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:33.244765 (+ 726us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:33.244769 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:33.244770 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:00:33.244977 (+ 207us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:33.245024 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:00:33.245042 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:33.245117 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:33.245186 (+ 69us) server_negotiation.cc:299] Negotiation successful
0926 15:00:33.245262 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":160,"thread_start_us":86,"threads_started":1}
I20260926 15:00:33.246434 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:00:33.251273 13795 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:00:33.251989 13795 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:00:33.259182 13795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ba957e47869460b838fc1ec9f4be9e8: Generated new TSK 0
I20260926 15:00:33.259423 13795 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 15:00:33.294184 13813 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260926 15:00:33.294313 13813 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 15:00:33.294739 13787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:33.272003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53558 (local address 127.8.102.62:38213)
0926 15:00:33.272056 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:33.272058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:33.272239 (+ 181us) server_negotiation.cc:407] Connection header received
0926 15:00:33.272353 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:33.272354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:33.272364 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:33.272427 (+ 63us) server_negotiation.cc:226] Negotiated authn=JWT
0926 15:00:33.272626 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.273112 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.274259 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.274444 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.274498 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:33.274758 (+ 260us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 15:00:33.294355 (+ 19597us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34635/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 15:00:33.294597 (+ 242us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53558: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34635/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 15:00:33.294811 13787 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53558: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34635/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260926 15:00:33.295121 13812 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:33.271880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:38213 (local address 127.0.0.1:53558)
0926 15:00:33.272153 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:33.272165 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:33.272245 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:33.272450 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:33.272451 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:33.272453 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 15:00:33.272532 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:33.272534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.273192 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.273193 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:33.274168 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:33.274171 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.274527 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.274528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:33.274594 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:33.274619 (+ 25us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 15:00:33.294597 (+ 19978us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34635/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0926 15:00:33.294779 (+ 182us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.102.62:38213: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34635/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":110,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.8.102.62:38213: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34635/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260926 15:00:33.295640 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 13716
2026-09-26T15:00:33Z chronyd exiting
I20260926 15:00:33.331936 8600 test_util.cc:175] -----------------------------------------------
I20260926 15:00:33.332024 8600 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790434810362722-8600-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (238 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260926 15:00:33.332823 8600 mini_oidc.cc:152] Starting JWKS server
I20260926 15:00:33.333562 8600 webserver.cc:460] Webserver started at https://127.0.0.1:34153/ using document root <none> and password file <none>
I20260926 15:00:33.333607 8600 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 15:00:33.333750 8600 webserver.cc:460] Webserver started at http://127.0.0.1:39511/ using document root <none> and password file <none>
2026-09-26T15:00:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:00:33Z Disabled control of system clock
I20260926 15:00:33.341753 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:46737
--webserver_interface=127.8.102.62
--webserver_port=0
--builtin_ntp_servers=127.8.102.20:35779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:46737
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34153/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434810362722-8600-0/testchainca.cert with env {}
W20260926 15:00:33.422456 13819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:33.422740 13819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:33.422780 13819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:33.424012 13819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:33.424105 13819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:33.424125 13819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:33.424146 13819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:33.424173 13819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 15:00:33.424206 13819 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 15:00:33.424228 13819 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34153/jwks.json
I20260926 15:00:33.425876 13819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:35779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:46737
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:46737
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34153/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434810362722-8600-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13819
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:33.426179 13819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:33.426401 13819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:33.431520 13824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:33.431583 13825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:33.431651 13819 server_base.cc:1034] running on GCE node
W20260926 15:00:33.431763 13827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:33.432003 13819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:33.432358 13819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:33.433535 13819 hybrid_clock.cc:648] HybridClock initialized: now 1790434833433502 us; error 58 us; skew 500 ppm
I20260926 15:00:33.434899 13819 webserver.cc:460] Webserver started at http://127.8.102.62:41327/ using document root <none> and password file <none>
I20260926 15:00:33.435159 13819 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:33.435230 13819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:33.435379 13819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:33.436479 13819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "e77542c9169d4d23927d2fbd9f2034bf"
format_stamp: "Formatted at 2026-09-26 15:00:33 on dist-test-slave-jnhs"
server_key: "eb9bbecbd29da2185a45f9e28341cf0a"
server_key_iv: "cce89d3dd8f7d7523b68b815f59c72b7"
server_key_version: "encryptionkey@0"
I20260926 15:00:33.436852 13819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "e77542c9169d4d23927d2fbd9f2034bf"
format_stamp: "Formatted at 2026-09-26 15:00:33 on dist-test-slave-jnhs"
server_key: "eb9bbecbd29da2185a45f9e28341cf0a"
server_key_iv: "cce89d3dd8f7d7523b68b815f59c72b7"
server_key_version: "encryptionkey@0"
I20260926 15:00:33.438179 13819 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:00:33.439067 13833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:33.439249 13819 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 15:00:33.439359 13819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "e77542c9169d4d23927d2fbd9f2034bf"
format_stamp: "Formatted at 2026-09-26 15:00:33 on dist-test-slave-jnhs"
server_key: "eb9bbecbd29da2185a45f9e28341cf0a"
server_key_iv: "cce89d3dd8f7d7523b68b815f59c72b7"
server_key_version: "encryptionkey@0"
I20260926 15:00:33.439438 13819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:33.447928 13819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:33.448256 13819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:33.448407 13819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:33.452628 13819 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:46737
I20260926 15:00:33.452659 13885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:46737 every 8 connection(s)
I20260926 15:00:33.453078 13819 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:00:33.454066 13886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:33.456728 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 13819
I20260926 15:00:33.456869 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:00:33.457088 8600 external_mini_cluster.cc:1428] Setting key c1b194e1f8b78832706fd3c8a96be520
I20260926 15:00:33.457231 13886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf: Bootstrap starting.
I20260926 15:00:33.458302 13886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:33.458667 13886 log.cc:824] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:33.459547 13886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf: No bootstrap required, opened a new log
I20260926 15:00:33.462064 13886 raft_consensus.cc:348] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77542c9169d4d23927d2fbd9f2034bf" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 46737 } }
I20260926 15:00:33.462278 13886 raft_consensus.cc:374] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:33.462322 13886 raft_consensus.cc:729] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e77542c9169d4d23927d2fbd9f2034bf, State: Initialized, Role: FOLLOWER
I20260926 15:00:33.462472 13886 consensus_queue.cc:260] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77542c9169d4d23927d2fbd9f2034bf" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 46737 } }
I20260926 15:00:33.462560 13886 raft_consensus.cc:388] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:00:33.462608 13886 raft_consensus.cc:482] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:00:33.462683 13886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:33.463485 13886 raft_consensus.cc:504] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77542c9169d4d23927d2fbd9f2034bf" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 46737 } }
I20260926 15:00:33.463603 13886 leader_election.cc:302] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e77542c9169d4d23927d2fbd9f2034bf; no voters:
I20260926 15:00:33.463768 13886 leader_election.cc:288] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:00:33.464030 13886 sys_catalog.cc:564] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:33.464193 13891 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:33.464370 13891 raft_consensus.cc:686] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf [term 1 LEADER]: Becoming Leader. State: Replica: e77542c9169d4d23927d2fbd9f2034bf, State: Running, Role: LEADER
I20260926 15:00:33.464468 13891 consensus_queue.cc:237] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77542c9169d4d23927d2fbd9f2034bf" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 46737 } }
I20260926 15:00:33.464835 13892 sys_catalog.cc:455] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e77542c9169d4d23927d2fbd9f2034bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77542c9169d4d23927d2fbd9f2034bf" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 46737 } } }
I20260926 15:00:33.464920 13892 sys_catalog.cc:458] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:33.464870 13893 sys_catalog.cc:455] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader e77542c9169d4d23927d2fbd9f2034bf. Latest consensus state: current_term: 1 leader_uuid: "e77542c9169d4d23927d2fbd9f2034bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e77542c9169d4d23927d2fbd9f2034bf" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 46737 } } }
I20260926 15:00:33.465394 13893 sys_catalog.cc:458] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:33.466365 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:33.457560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45608 (local address 127.8.102.62:46737)
0926 15:00:33.457831 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:33.457844 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:33.458438 (+ 594us) server_negotiation.cc:407] Connection header received
0926 15:00:33.459197 (+ 759us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:33.459207 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:33.459296 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:33.459453 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:33.459999 (+ 546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.461431 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.463085 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.463738 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.464199 (+ 461us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:33.465422 (+ 1223us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:33.465427 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:33.465427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:00:33.465645 (+ 218us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:33.465688 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:00:33.465704 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:33.465770 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:33.466057 (+ 287us) server_negotiation.cc:299] Negotiation successful
0926 15:00:33.466152 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":110,"threads_started":1}
W20260926 15:00:33.466739 13906 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:00:33.466796 13906 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:00:33.466851 13900 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:00:33.466996 13900 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:00:33.467365 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:00:33.468719 13900 catalog_manager.cc:1347] Generated new cluster ID: 912ef514dcab46618c3df1d920d9978e
I20260926 15:00:33.468772 13900 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:00:33.481637 13900 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:00:33.482609 13900 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:00:33.490298 13900 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e77542c9169d4d23927d2fbd9f2034bf: Generated new TSK 0
I20260926 15:00:33.490752 13900 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260926 15:00:33.524240 13916 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260926 15:00:33.524336 13916 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260926 15:00:33.524945 13889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:33.498390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45622 (local address 127.8.102.62:46737)
0926 15:00:33.498449 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:33.498451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:33.498460 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:33.498504 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:33.498504 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:33.498515 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:33.498609 (+ 94us) server_negotiation.cc:226] Negotiated authn=JWT
0926 15:00:33.498805 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.499620 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.500952 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.501263 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.501336 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:33.501589 (+ 253us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 15:00:33.524644 (+ 23055us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34153/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 15:00:33.524847 (+ 203us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:45622: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34153/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":25}
W20260926 15:00:33.525024 13915 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:33.497884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:46737 (local address 127.0.0.1:45622)
0926 15:00:33.498021 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:33.498034 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:33.498112 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:33.498611 (+ 499us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:33.498612 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:33.498615 (+ 3us) client_negotiation.cc:189] Negotiated authn=JWT
0926 15:00:33.498724 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:33.498728 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.499723 (+ 995us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.499725 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:33.500839 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:33.500843 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.501388 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.501390 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:33.501454 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:33.501461 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 15:00:33.524783 (+ 23322us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34153/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0926 15:00:33.524913 (+ 130us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.102.62:46737: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34153/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":77,"threads_started":1}
I20260926 15:00:33.525303 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 13819
2026-09-26T15:00:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (231 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260926 15:00:33.564355 8600 mini_oidc.cc:152] Starting JWKS server
I20260926 15:00:33.565275 8600 webserver.cc:460] Webserver started at https://127.0.0.1:39067/ using document root <none> and password file <none>
I20260926 15:00:33.565366 8600 mini_oidc.cc:189] Starting OIDC Discovery server
I20260926 15:00:33.565594 8600 webserver.cc:460] Webserver started at http://127.0.0.1:42181/ using document root <none> and password file <none>
2026-09-26T15:00:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:00:33Z Disabled control of system clock
I20260926 15:00:33.573977 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:45671
--webserver_interface=127.8.102.62
--webserver_port=0
--builtin_ntp_servers=127.8.102.20:46095
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:45671
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39067/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 15:00:33.659899 13922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:33.660123 13922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:33.660147 13922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:33.661391 13922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:33.661444 13922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:33.661458 13922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:33.661470 13922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:33.661484 13922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 15:00:33.661500 13922 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 15:00:33.661556 13922 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:39067/jwks.json
I20260926 15:00:33.663035 13922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46095
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:45671
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:45671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:39067/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.13922
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:33.663321 13922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:33.663653 13922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:33.668915 13930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:33.668901 13928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:33.668901 13927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:33.669468 13922 server_base.cc:1034] running on GCE node
I20260926 15:00:33.669723 13922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:33.669988 13922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:33.671149 13922 hybrid_clock.cc:648] HybridClock initialized: now 1790434833671124 us; error 39 us; skew 500 ppm
I20260926 15:00:33.672605 13922 webserver.cc:460] Webserver started at http://127.8.102.62:35401/ using document root <none> and password file <none>
I20260926 15:00:33.672907 13922 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:33.672991 13922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:33.673169 13922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:33.674448 13922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "24e6d9e326034e8b92091d23ec75ae84"
format_stamp: "Formatted at 2026-09-26 15:00:33 on dist-test-slave-jnhs"
server_key: "5f81fab14b3cf883c51015414a84c7f9"
server_key_iv: "22e6bd2cd7c1328700df1d0a3f2dc19f"
server_key_version: "encryptionkey@0"
I20260926 15:00:33.674865 13922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "24e6d9e326034e8b92091d23ec75ae84"
format_stamp: "Formatted at 2026-09-26 15:00:33 on dist-test-slave-jnhs"
server_key: "5f81fab14b3cf883c51015414a84c7f9"
server_key_iv: "22e6bd2cd7c1328700df1d0a3f2dc19f"
server_key_version: "encryptionkey@0"
I20260926 15:00:33.676434 13922 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:33.677343 13936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:33.677536 13922 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:33.677646 13922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "24e6d9e326034e8b92091d23ec75ae84"
format_stamp: "Formatted at 2026-09-26 15:00:33 on dist-test-slave-jnhs"
server_key: "5f81fab14b3cf883c51015414a84c7f9"
server_key_iv: "22e6bd2cd7c1328700df1d0a3f2dc19f"
server_key_version: "encryptionkey@0"
I20260926 15:00:33.677736 13922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:33.700176 13922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:33.700593 13922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:33.700773 13922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:33.705152 13922 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:45671
I20260926 15:00:33.705204 13988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:45671 every 8 connection(s)
I20260926 15:00:33.705586 13922 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:00:33.706334 13989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:33.708971 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 13922
I20260926 15:00:33.709084 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:00:33.709264 8600 external_mini_cluster.cc:1428] Setting key 75abd09b6116d2a9ef3a3f6b60aeedd3
I20260926 15:00:33.709538 13989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84: Bootstrap starting.
I20260926 15:00:33.710264 13989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:33.710567 13989 log.cc:824] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:33.711570 13989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84: No bootstrap required, opened a new log
I20260926 15:00:33.714021 13989 raft_consensus.cc:348] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e6d9e326034e8b92091d23ec75ae84" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45671 } }
I20260926 15:00:33.714212 13989 raft_consensus.cc:374] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:33.714259 13989 raft_consensus.cc:729] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24e6d9e326034e8b92091d23ec75ae84, State: Initialized, Role: FOLLOWER
I20260926 15:00:33.714399 13989 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e6d9e326034e8b92091d23ec75ae84" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45671 } }
I20260926 15:00:33.714506 13989 raft_consensus.cc:388] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:00:33.714548 13989 raft_consensus.cc:482] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:00:33.714607 13989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:33.715399 13989 raft_consensus.cc:504] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e6d9e326034e8b92091d23ec75ae84" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45671 } }
I20260926 15:00:33.715543 13989 leader_election.cc:302] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 24e6d9e326034e8b92091d23ec75ae84; no voters:
I20260926 15:00:33.715761 13989 leader_election.cc:288] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:00:33.715814 13994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:33.715991 13994 raft_consensus.cc:686] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84 [term 1 LEADER]: Becoming Leader. State: Replica: 24e6d9e326034e8b92091d23ec75ae84, State: Running, Role: LEADER
I20260926 15:00:33.716063 13989 sys_catalog.cc:564] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:33.716094 13994 consensus_queue.cc:237] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e6d9e326034e8b92091d23ec75ae84" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45671 } }
I20260926 15:00:33.716457 13992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:33.709828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49436 (local address 127.8.102.62:45671)
0926 15:00:33.710062 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:33.710072 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:33.710139 (+ 67us) server_negotiation.cc:407] Connection header received
0926 15:00:33.710682 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:33.710693 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:33.710787 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:33.710968 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:33.711409 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.712450 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.713667 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.714163 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.714384 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:33.715199 (+ 815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:33.715204 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:33.715204 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:00:33.715405 (+ 201us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:33.715444 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:00:33.715456 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:33.715709 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:33.715811 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 15:00:33.716084 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":93,"threads_started":1}
I20260926 15:00:33.716529 13995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 24e6d9e326034e8b92091d23ec75ae84. Latest consensus state: current_term: 1 leader_uuid: "24e6d9e326034e8b92091d23ec75ae84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e6d9e326034e8b92091d23ec75ae84" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45671 } } }
I20260926 15:00:33.716535 13996 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "24e6d9e326034e8b92091d23ec75ae84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "24e6d9e326034e8b92091d23ec75ae84" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45671 } } }
I20260926 15:00:33.716638 13995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:33.716693 13996 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:33.716887 13998 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:00:33.717621 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:00:33.718324 13998 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:00:33.720098 13998 catalog_manager.cc:1347] Generated new cluster ID: 187ef903b20248a6abcee58b222cf0f1
I20260926 15:00:33.720162 13998 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:00:33.733121 13998 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:00:33.733943 13998 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:00:33.738207 13998 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 24e6d9e326034e8b92091d23ec75ae84: Generated new TSK 0
I20260926 15:00:33.738425 13998 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:00:33.753139 13992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:33.749263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49446 (local address 127.8.102.62:45671)
0926 15:00:33.749313 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:33.749315 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:33.749418 (+ 103us) server_negotiation.cc:407] Connection header received
0926 15:00:33.749497 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:33.749498 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:33.749508 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:33.749615 (+ 107us) server_negotiation.cc:226] Negotiated authn=JWT
0926 15:00:33.749783 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.750295 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.751467 (+ 1172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.751669 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.751727 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:33.751921 (+ 194us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 15:00:33.752927 (+ 1006us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39067/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 15:00:33.753059 (+ 132us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49446: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39067/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":26}
W20260926 15:00:33.753206 13992 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49446: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39067/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260926 15:00:33.753250 14018 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:33.749136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45671 (local address 127.0.0.1:49446)
0926 15:00:33.749324 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:33.749336 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:33.749412 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:33.749624 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:33.749625 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:33.749627 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 15:00:33.749693 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:33.749696 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.750387 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.750388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:33.751370 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:33.751373 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.751752 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.751753 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:33.751818 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:33.751825 (+ 7us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 15:00:33.753034 (+ 1209us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39067/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0926 15:00:33.753151 (+ 117us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.102.62:45671: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:39067/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":119,"threads_started":1}
I20260926 15:00:33.753597 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 13922
E20260926 15:00:33.764484 14019 webserver.cc:533] Webserver: sslize failed:
E20260926 15:00:33.764595 14019 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-26T15:00:33Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (226 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-26T15:00:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:00:33Z Disabled control of system clock
I20260926 15:00:33.799273 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:38047
--webserver_interface=127.8.102.62
--webserver_port=0
--builtin_ntp_servers=127.8.102.20:37919
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:38047
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434810362722-8600-0/testchainca.cert with env {}
W20260926 15:00:33.882606 14023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:33.882854 14023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:33.882879 14023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:33.884153 14023 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:33.884205 14023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:33.884218 14023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:33.884232 14023 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:33.884244 14023 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260926 15:00:33.884258 14023 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260926 15:00:33.884268 14023 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260926 15:00:33.885862 14023 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:37919
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:38047
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:38047
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434810362722-8600-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:33.886089 14023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:33.886261 14023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:33.891582 14029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:33.891753 14031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:33.891767 14028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:33.892170 14023 server_base.cc:1034] running on GCE node
I20260926 15:00:33.892408 14023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:33.892762 14023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:33.893977 14023 hybrid_clock.cc:648] HybridClock initialized: now 1790434833893965 us; error 42 us; skew 500 ppm
I20260926 15:00:33.895597 14023 webserver.cc:460] Webserver started at http://127.8.102.62:40425/ using document root <none> and password file <none>
I20260926 15:00:33.895908 14023 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:33.896001 14023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:33.896173 14023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:33.897410 14023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "a101464a7d154032a7a5f969f828c707"
format_stamp: "Formatted at 2026-09-26 15:00:33 on dist-test-slave-jnhs"
server_key: "a06098f92cec22439f2affa1b81e852a"
server_key_iv: "6904b2f1457274b2289edb50d6f852b2"
server_key_version: "encryptionkey@0"
I20260926 15:00:33.897855 14023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "a101464a7d154032a7a5f969f828c707"
format_stamp: "Formatted at 2026-09-26 15:00:33 on dist-test-slave-jnhs"
server_key: "a06098f92cec22439f2affa1b81e852a"
server_key_iv: "6904b2f1457274b2289edb50d6f852b2"
server_key_version: "encryptionkey@0"
I20260926 15:00:33.899514 14023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260926 15:00:33.900516 14037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:33.900784 14023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:33.900897 14023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "a101464a7d154032a7a5f969f828c707"
format_stamp: "Formatted at 2026-09-26 15:00:33 on dist-test-slave-jnhs"
server_key: "a06098f92cec22439f2affa1b81e852a"
server_key_iv: "6904b2f1457274b2289edb50d6f852b2"
server_key_version: "encryptionkey@0"
I20260926 15:00:33.901011 14023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:33.916999 14023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:33.917389 14023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:33.917580 14023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:33.922108 14023 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:38047
I20260926 15:00:33.922154 14089 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:38047 every 8 connection(s)
I20260926 15:00:33.922616 14023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:00:33.923733 14090 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:33.924018 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 14023
I20260926 15:00:33.924136 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:00:33.924352 8600 external_mini_cluster.cc:1428] Setting key 8a4ab2d306c60869b500d58b9234af00
I20260926 15:00:33.926473 14090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707: Bootstrap starting.
I20260926 15:00:33.927143 14090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:33.927426 14090 log.cc:824] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:33.928158 14090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707: No bootstrap required, opened a new log
I20260926 15:00:33.930130 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:33.924909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57142 (local address 127.8.102.62:38047)
0926 15:00:33.925132 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:33.925141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:33.925167 (+ 26us) server_negotiation.cc:407] Connection header received
0926 15:00:33.925588 (+ 421us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:33.925595 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:33.925658 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:33.925819 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:33.926060 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.927074 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.927970 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.928489 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.928633 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:33.929458 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:33.929462 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:33.929463 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:00:33.929639 (+ 176us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:33.929678 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:00:33.929689 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:33.929771 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:33.929859 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 15:00:33.929950 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":83,"threads_started":1}
I20260926 15:00:33.930389 14090 raft_consensus.cc:348] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a101464a7d154032a7a5f969f828c707" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38047 } }
I20260926 15:00:33.930579 14090 raft_consensus.cc:374] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:33.930621 14090 raft_consensus.cc:729] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a101464a7d154032a7a5f969f828c707, State: Initialized, Role: FOLLOWER
I20260926 15:00:33.930779 14090 consensus_queue.cc:260] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a101464a7d154032a7a5f969f828c707" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38047 } }
I20260926 15:00:33.930886 14090 raft_consensus.cc:388] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:00:33.930919 14090 raft_consensus.cc:482] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:00:33.930961 14090 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:33.931371 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:00:33.931735 14090 raft_consensus.cc:504] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a101464a7d154032a7a5f969f828c707" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38047 } }
I20260926 15:00:33.931828 14090 leader_election.cc:302] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a101464a7d154032a7a5f969f828c707; no voters:
I20260926 15:00:33.932039 14090 leader_election.cc:288] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:00:33.932106 14095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:33.932212 14095 raft_consensus.cc:686] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707 [term 1 LEADER]: Becoming Leader. State: Replica: a101464a7d154032a7a5f969f828c707, State: Running, Role: LEADER
I20260926 15:00:33.932288 14095 consensus_queue.cc:237] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a101464a7d154032a7a5f969f828c707" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38047 } }
I20260926 15:00:33.932312 14090 sys_catalog.cc:564] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:33.932611 14096 sys_catalog.cc:455] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a101464a7d154032a7a5f969f828c707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a101464a7d154032a7a5f969f828c707" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38047 } } }
I20260926 15:00:33.932719 14096 sys_catalog.cc:458] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:33.932891 14097 sys_catalog.cc:455] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a101464a7d154032a7a5f969f828c707. Latest consensus state: current_term: 1 leader_uuid: "a101464a7d154032a7a5f969f828c707" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a101464a7d154032a7a5f969f828c707" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38047 } } }
I20260926 15:00:33.932976 14097 sys_catalog.cc:458] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:33.932924 14099 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:00:33.933496 14099 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:00:33.935184 14099 catalog_manager.cc:1347] Generated new cluster ID: 333deafa18044e91b0816c96b1d9215b
I20260926 15:00:33.935242 14099 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:00:33.963104 14099 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:00:33.963862 14099 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:00:33.972218 14099 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a101464a7d154032a7a5f969f828c707: Generated new TSK 0
I20260926 15:00:33.972455 14099 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:00:34.021624 14093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:33.985603 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57158 (local address 127.8.102.62:38047)
0926 15:00:33.985657 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:33.985659 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:33.985760 (+ 101us) server_negotiation.cc:407] Connection header received
0926 15:00:33.985811 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:33.985812 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:33.985828 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:33.985919 (+ 91us) server_negotiation.cc:226] Negotiated authn=JWT
0926 15:00:33.986062 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.986878 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.987820 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.988152 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.988228 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:33.988427 (+ 199us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0926 15:00:34.021291 (+ 32864us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 15:00:34.021538 (+ 247us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:57158: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":28}
W20260926 15:00:34.021715 14093 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:57158: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260926 15:00:34.021764 14119 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:33.985509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:38047 (local address 127.0.0.1:57158)
0926 15:00:33.985673 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:33.985685 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:33.985753 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:33.985924 (+ 171us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:33.985924 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:33.985926 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0926 15:00:33.985987 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:33.985990 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.987011 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.987013 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:33.987722 (+ 709us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:33.987724 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:33.988250 (+ 526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:33.988251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:33.988304 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:33.988310 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0926 15:00:34.021495 (+ 33185us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0926 15:00:34.021639 (+ 144us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.8.102.62:38047: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":108,"threads_started":1}
I20260926 15:00:34.022166 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 14023
2026-09-26T15:00:34Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (248 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (92 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (91 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:00:34 dist-test-slave-jnhs krb5kdc[14131](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:00:34 dist-test-slave-jnhs krb5kdc[14131](info): set up 2 sockets
Sep 26 15:00:34 dist-test-slave-jnhs krb5kdc[14131](info): commencing operation
krb5kdc: starting...
W20260926 15:00:35.250976 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.026s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:00:35 dist-test-slave-jnhs krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:00:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:00:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:00:35.379302 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:44261
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:42899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:44261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:35.470288 14144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:35.470590 14144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:35.470639 14144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:35.472005 14144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:35.472131 14144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:35.472158 14144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:35.472184 14144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:35.472216 14144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:00:35.473934 14144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:42899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:44261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:44261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14144
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:35.474262 14144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:35.474526 14144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:35.479939 14149 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:35.479936 14150 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:35.480202 14144 server_base.cc:1034] running on GCE node
W20260926 15:00:35.479960 14152 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:35.480580 14144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:35.481055 14144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:35.482290 14144 hybrid_clock.cc:648] HybridClock initialized: now 1790434835482274 us; error 34 us; skew 500 ppm
Sep 26 15:00:35 dist-test-slave-jnhs krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434835, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:35.485152 14144 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:00:35.485657 14144 webserver.cc:460] Webserver started at http://127.8.102.62:37521/ using document root <none> and password file <none>
I20260926 15:00:35.485908 14144 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:35.485954 14144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:35.486109 14144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:35.487280 14144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "7ff2006347384485b94c5114e9f7d860"
format_stamp: "Formatted at 2026-09-26 15:00:35 on dist-test-slave-jnhs"
server_key: "a204e7994e0eb6722cb46afe050a1ec4"
server_key_iv: "a6e8907c6e2f97327017d28fd5aa4b52"
server_key_version: "encryptionkey@0"
I20260926 15:00:35.487658 14144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "7ff2006347384485b94c5114e9f7d860"
format_stamp: "Formatted at 2026-09-26 15:00:35 on dist-test-slave-jnhs"
server_key: "a204e7994e0eb6722cb46afe050a1ec4"
server_key_iv: "a6e8907c6e2f97327017d28fd5aa4b52"
server_key_version: "encryptionkey@0"
I20260926 15:00:35.489060 14144 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:00:35.489961 14159 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:35.490235 14144 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:35.490350 14144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "7ff2006347384485b94c5114e9f7d860"
format_stamp: "Formatted at 2026-09-26 15:00:35 on dist-test-slave-jnhs"
server_key: "a204e7994e0eb6722cb46afe050a1ec4"
server_key_iv: "a6e8907c6e2f97327017d28fd5aa4b52"
server_key_version: "encryptionkey@0"
I20260926 15:00:35.490445 14144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:35.499100 14144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:35.501566 14144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:35.501744 14144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:35.506392 14144 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:44261
I20260926 15:00:35.506471 14211 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:44261 every 8 connection(s)
I20260926 15:00:35.506879 14144 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:00:35.507845 14212 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:35.510447 14212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860: Bootstrap starting.
I20260926 15:00:35.511145 14212 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:35.511477 14212 log.cc:824] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:35.512331 14212 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860: No bootstrap required, opened a new log
I20260926 15:00:35.514034 14212 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff2006347384485b94c5114e9f7d860" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44261 } }
I20260926 15:00:35.514184 14212 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:35.514207 14212 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ff2006347384485b94c5114e9f7d860, State: Initialized, Role: FOLLOWER
I20260926 15:00:35.514375 14212 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff2006347384485b94c5114e9f7d860" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44261 } }
I20260926 15:00:35.514451 14212 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:00:35.514489 14212 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:00:35.514518 14212 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:35.515023 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 14144
I20260926 15:00:35.515111 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:00:35.515204 14212 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff2006347384485b94c5114e9f7d860" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44261 } }
I20260926 15:00:35.515328 14212 leader_election.cc:302] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ff2006347384485b94c5114e9f7d860; no voters:
I20260926 15:00:35.515349 8600 external_mini_cluster.cc:1428] Setting key 882ecdb364249c58069e40d42f2034ee
I20260926 15:00:35.515553 14212 leader_election.cc:288] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:00:35.515590 14215 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:35.515853 14212 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:35.515929 14215 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860 [term 1 LEADER]: Becoming Leader. State: Replica: 7ff2006347384485b94c5114e9f7d860, State: Running, Role: LEADER
I20260926 15:00:35.516016 14215 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff2006347384485b94c5114e9f7d860" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44261 } }
I20260926 15:00:35.516369 14216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ff2006347384485b94c5114e9f7d860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff2006347384485b94c5114e9f7d860" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44261 } } }
I20260926 15:00:35.516602 14216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:35.516369 14218 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ff2006347384485b94c5114e9f7d860. Latest consensus state: current_term: 1 leader_uuid: "7ff2006347384485b94c5114e9f7d860" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ff2006347384485b94c5114e9f7d860" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44261 } } }
I20260926 15:00:35.516858 14218 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:35.517177 14226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:00:35.517724 14226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:00:35.519433 14226 catalog_manager.cc:1347] Generated new cluster ID: 222a3affea56491886e3465679b485d3
I20260926 15:00:35.519488 14226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 15:00:35 dist-test-slave-jnhs krb5kdc[14131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:35.527822 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:35.516086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36160 (local address 127.8.102.62:44261)
0926 15:00:35.516450 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:35.516462 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:35.516505 (+ 43us) server_negotiation.cc:407] Connection header received
0926 15:00:35.516646 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:35.516656 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:35.516720 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:35.517733 (+ 1013us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:35.517895 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:35.519267 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:35.520364 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:35.521035 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:35.521159 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:35.524074 (+ 2915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:35.524084 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:35.524084 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:35.524098 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:35.526035 (+ 1937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:35.526474 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:35.526477 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:35.526478 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:35.526591 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:35.526835 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:35.526836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:35.526836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:35.527151 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:35.527297 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:35.527446 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 15:00:35.527583 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":76,"threads_started":1}
I20260926 15:00:35.529223 14226 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:00:35.529995 14226 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:00:35.532735 14226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ff2006347384485b94c5114e9f7d860: Generated new TSK 0
I20260926 15:00:35.532910 14226 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
I20260926 15:00:35.562089 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44261
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:42899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:35.648842 14238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:35.649106 14238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:35.649150 14238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:35.650413 14238 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:35.650487 14238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:35.650537 14238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:00:35.652190 14238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:42899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14238
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:35.652499 14238 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:35.652757 14238 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:35.658365 14246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:35.658365 14244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:35.658367 14243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:35.658888 14238 server_base.cc:1034] running on GCE node
I20260926 15:00:35.659113 14238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:35.659340 14238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:35.660508 14238 hybrid_clock.cc:648] HybridClock initialized: now 1790434835660491 us; error 40 us; skew 500 ppm
Sep 26 15:00:35 dist-test-slave-jnhs krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434835, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:35.663311 14238 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:00:35.663800 14238 webserver.cc:460] Webserver started at http://127.8.102.1:35945/ using document root <none> and password file <none>
I20260926 15:00:35.664038 14238 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:35.664108 14238 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:35.664255 14238 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:35.665362 14238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "76f9d6737f674953a1c9ba6346b622e0"
format_stamp: "Formatted at 2026-09-26 15:00:35 on dist-test-slave-jnhs"
server_key: "c29304b088bb519d51ab6664c2a94e4a"
server_key_iv: "99390d368abb5c552dafe79222da8d05"
server_key_version: "encryptionkey@0"
I20260926 15:00:35.665761 14238 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "76f9d6737f674953a1c9ba6346b622e0"
format_stamp: "Formatted at 2026-09-26 15:00:35 on dist-test-slave-jnhs"
server_key: "c29304b088bb519d51ab6664c2a94e4a"
server_key_iv: "99390d368abb5c552dafe79222da8d05"
server_key_version: "encryptionkey@0"
I20260926 15:00:35.667289 14238 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:00:35.668111 14253 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:35.668322 14238 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:35.668428 14238 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "76f9d6737f674953a1c9ba6346b622e0"
format_stamp: "Formatted at 2026-09-26 15:00:35 on dist-test-slave-jnhs"
server_key: "c29304b088bb519d51ab6664c2a94e4a"
server_key_iv: "99390d368abb5c552dafe79222da8d05"
server_key_version: "encryptionkey@0"
I20260926 15:00:35.668504 14238 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:35.686009 14238 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:35.688498 14238 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:35.688691 14238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:35.688957 14238 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:35.689277 14238 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:35.689328 14238 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:35.689364 14238 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:35.689406 14238 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:35.696086 14238 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:46735
I20260926 15:00:35.696177 14366 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:46735 every 8 connection(s)
I20260926 15:00:35.696583 14238 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:00:35.698063 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 14238
I20260926 15:00:35.698199 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:00:35.698418 8600 external_mini_cluster.cc:1428] Setting key e8b92e9aa2917bb77b814c4ee8836460
Sep 26 15:00:35 dist-test-slave-jnhs krb5kdc[14131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434835, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.102.2@KRBTEST.COM; defaulting to no policy
I20260926 15:00:35.708151 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:35.697542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:49681 (local address 127.8.102.62:44261)
0926 15:00:35.697600 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:35.697603 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:35.697933 (+ 330us) server_negotiation.cc:407] Connection header received
0926 15:00:35.699161 (+ 1228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:35.699162 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:35.699178 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:35.699261 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:35.699971 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:35.700488 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:35.701565 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:35.701790 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:35.701880 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:35.704681 (+ 2801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:35.704683 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:35.704683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:35.704692 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:35.706375 (+ 1683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:35.706998 (+ 623us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:35.707000 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:35.707000 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:35.707092 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:35.707397 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:35.707398 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:35.707399 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:35.707535 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:35.707614 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:35.707939 (+ 325us) server_negotiation.cc:299] Negotiation successful
0926 15:00:35.708050 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 15:00:35.708362 14369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:35.697593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44261 (local address 127.8.102.1:49681)
0926 15:00:35.697755 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:35.697815 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:35.699026 (+ 1211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:35.699343 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:35.699347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:35.699630 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:35.699839 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:35.699844 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:35.700589 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:35.700590 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:35.701435 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:35.701437 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:35.701890 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:35.701891 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:35.702029 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:35.702602 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:35.702612 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:35.704535 (+ 1923us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:35.706535 (+ 2000us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:35.706544 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:35.706547 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:35.706901 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:35.707171 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:35.707172 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:35.707173 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:35.707313 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:35.707605 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:35.707613 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:35.707783 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 15:00:35.707902 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 15:00:35.708090 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":80,"threads_started":1}
I20260926 15:00:35.708956 14367 heartbeater.cc:347] Connected to a master server at 127.8.102.62:44261
I20260926 15:00:35.709108 14367 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:35.709316 14367 heartbeater.cc:511] Master 127.8.102.62:44261 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.2@KRBTEST.COM" created.
I20260926 15:00:35.709961 14176 ts_manager.cc:194] Registered new tserver with Master: 76f9d6737f674953a1c9ba6346b622e0 (127.8.102.1:46735)
I20260926 15:00:35.710714 14176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:49681
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
I20260926 15:00:35.726437 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44261
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:42899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:35.810567 14372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:35.810882 14372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:35.810916 14372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:35.812139 14372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:35.812184 14372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:35.812212 14372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:00:35.813753 14372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:42899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14372
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:35.814021 14372 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:35.814260 14372 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:35.819921 14378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:35.819923 14377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:35.819939 14380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:35.820218 14372 server_base.cc:1034] running on GCE node
I20260926 15:00:35.820444 14372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:35.820699 14372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:35.821869 14372 hybrid_clock.cc:648] HybridClock initialized: now 1790434835821859 us; error 34 us; skew 500 ppm
Sep 26 15:00:35 dist-test-slave-jnhs krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434835, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:35.824424 14372 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:00:35.824904 14372 webserver.cc:460] Webserver started at http://127.8.102.2:39333/ using document root <none> and password file <none>
I20260926 15:00:35.825088 14372 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:35.825130 14372 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:35.825222 14372 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:35.826215 14372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "a67e174d164f42b38f6a81692c124b04"
format_stamp: "Formatted at 2026-09-26 15:00:35 on dist-test-slave-jnhs"
server_key: "68d91f3a55c6ab8a7832cb69f2f29a8f"
server_key_iv: "894ed78b69c569fa501e0299057a5e20"
server_key_version: "encryptionkey@0"
I20260926 15:00:35.826512 14372 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "a67e174d164f42b38f6a81692c124b04"
format_stamp: "Formatted at 2026-09-26 15:00:35 on dist-test-slave-jnhs"
server_key: "68d91f3a55c6ab8a7832cb69f2f29a8f"
server_key_iv: "894ed78b69c569fa501e0299057a5e20"
server_key_version: "encryptionkey@0"
I20260926 15:00:35.827873 14372 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:35.828769 14387 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:35.829061 14372 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:35.829196 14372 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "a67e174d164f42b38f6a81692c124b04"
format_stamp: "Formatted at 2026-09-26 15:00:35 on dist-test-slave-jnhs"
server_key: "68d91f3a55c6ab8a7832cb69f2f29a8f"
server_key_iv: "894ed78b69c569fa501e0299057a5e20"
server_key_version: "encryptionkey@0"
I20260926 15:00:35.829331 14372 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:35.842938 14372 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:35.845554 14372 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:35.845763 14372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:35.846035 14372 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:35.846377 14372 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:35.846437 14372 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:35.846498 14372 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:35.846529 14372 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:35.852965 14372 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:46457
I20260926 15:00:35.853003 14500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:46457 every 8 connection(s)
I20260926 15:00:35.853371 14372 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
Sep 26 15:00:35 dist-test-slave-jnhs krb5kdc[14131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434835, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:35.861929 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 14372
I20260926 15:00:35.862025 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:00:35.862234 8600 external_mini_cluster.cc:1428] Setting key 42f335107fec81a05218e143d8d8b0a5
I20260926 15:00:35.863601 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:35.854171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:56867 (local address 127.8.102.62:44261)
0926 15:00:35.854242 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:35.854244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:35.854446 (+ 202us) server_negotiation.cc:407] Connection header received
0926 15:00:35.855478 (+ 1032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:35.855479 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:35.855492 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:35.855616 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:35.856283 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:35.856819 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:35.857851 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:35.858053 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:35.858141 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:35.860444 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:35.860446 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:35.860446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:35.860454 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:35.861911 (+ 1457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:35.862477 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:35.862479 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:35.862480 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:35.862549 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:35.862864 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:35.862866 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:35.862866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:35.862993 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:35.863071 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:35.863388 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 15:00:35.863502 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 15:00:35.863862 14503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:35.854126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44261 (local address 127.8.102.2:56867)
0926 15:00:35.854284 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:35.854326 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:35.855358 (+ 1032us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:35.855699 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:35.855703 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:35.855980 (+ 277us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:35.856183 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:35.856188 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:35.856927 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:35.856928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:35.857756 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:35.857758 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:35.858151 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:35.858153 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:35.858287 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:35.858874 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:35.858883 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:35.860336 (+ 1453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:35.862041 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:35.862049 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:35.862052 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:35.862385 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:35.862623 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:35.862624 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:35.862625 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:35.862779 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:35.863074 (+ 295us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:35.863081 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:35.863238 (+ 157us) client_negotiation.cc:769] Sending connection context
0926 15:00:35.863351 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 15:00:35.863531 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
I20260926 15:00:35.864362 14501 heartbeater.cc:347] Connected to a master server at 127.8.102.62:44261
I20260926 15:00:35.864472 14501 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:35.864671 14501 heartbeater.cc:511] Master 127.8.102.62:44261 requested a full tablet report, sending...
I20260926 15:00:35.865068 14176 ts_manager.cc:194] Registered new tserver with Master: a67e174d164f42b38f6a81692c124b04 (127.8.102.2:46457)
I20260926 15:00:35.865494 14176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:56867
WARNING: no policy specified for kudu/127.8.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
I20260926 15:00:35.891104 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44261
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:42899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:35.981468 14506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:35.981738 14506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:35.981778 14506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:35.983180 14506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:35.983264 14506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:35.983309 14506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:00:35.984863 14506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:42899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:35.985176 14506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:35.985421 14506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:35.990578 14511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:35.990603 14512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:35.990871 14506 server_base.cc:1034] running on GCE node
W20260926 15:00:35.990583 14514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:35.991156 14506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:35.991389 14506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:35.992611 14506 hybrid_clock.cc:648] HybridClock initialized: now 1790434835992587 us; error 31 us; skew 500 ppm
Sep 26 15:00:35 dist-test-slave-jnhs krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434835, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:35.995653 14506 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:00:35.996160 14506 webserver.cc:460] Webserver started at http://127.8.102.3:40453/ using document root <none> and password file <none>
I20260926 15:00:35.996404 14506 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:35.996521 14506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:35.996675 14506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:35.997740 14506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "4a04fcf168034df2b21ceed31cd04081"
format_stamp: "Formatted at 2026-09-26 15:00:35 on dist-test-slave-jnhs"
server_key: "d3bb2bdb9544237ba4009458ce0f636c"
server_key_iv: "5f82a20f4ef78601c000f9b13617606e"
server_key_version: "encryptionkey@0"
I20260926 15:00:35.998118 14506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "4a04fcf168034df2b21ceed31cd04081"
format_stamp: "Formatted at 2026-09-26 15:00:35 on dist-test-slave-jnhs"
server_key: "d3bb2bdb9544237ba4009458ce0f636c"
server_key_iv: "5f82a20f4ef78601c000f9b13617606e"
server_key_version: "encryptionkey@0"
I20260926 15:00:35.999657 14506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 15:00:36.000533 14521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:36.000717 14506 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:36.000829 14506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "4a04fcf168034df2b21ceed31cd04081"
format_stamp: "Formatted at 2026-09-26 15:00:35 on dist-test-slave-jnhs"
server_key: "d3bb2bdb9544237ba4009458ce0f636c"
server_key_iv: "5f82a20f4ef78601c000f9b13617606e"
server_key_version: "encryptionkey@0"
I20260926 15:00:36.000921 14506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:36.011117 14506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:36.013688 14506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:36.013852 14506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:36.014134 14506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:36.014501 14506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:36.014539 14506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:36.014562 14506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:36.014575 14506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:36.021211 14506 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:45101
I20260926 15:00:36.021250 14634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:45101 every 8 connection(s)
I20260926 15:00:36.021613 14506 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:00:36.027390 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 14506
I20260926 15:00:36.027503 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:00:36.027824 8600 external_mini_cluster.cc:1428] Setting key f99101f1bf6e09518e2abe72e4254946
Sep 26 15:00:36 dist-test-slave-jnhs krb5kdc[14131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434835, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:36.031719 14219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:36.022310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:37297 (local address 127.8.102.62:44261)
0926 15:00:36.022355 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:36.022357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:36.022585 (+ 228us) server_negotiation.cc:407] Connection header received
0926 15:00:36.023593 (+ 1008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:36.023595 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:36.023606 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:36.023663 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:36.024270 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:36.024780 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:36.025857 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:36.026047 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:36.026102 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:36.028758 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:36.028760 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:36.028760 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:36.028768 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:36.030165 (+ 1397us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:36.030687 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:36.030689 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:36.030689 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:36.030755 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:36.031031 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:36.031032 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:36.031032 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:36.031156 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:36.031234 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:36.031539 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 15:00:36.031627 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 15:00:36.031942 14637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:36.022312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44261 (local address 127.8.102.3:37297)
0926 15:00:36.022456 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:36.022499 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:36.023468 (+ 969us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:36.023724 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:36.023727 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:36.023992 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:36.024189 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:36.024193 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:36.024895 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:36.024896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:36.025775 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:36.025778 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:36.026123 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:36.026124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:36.026225 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:36.026880 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:36.026889 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:36.028643 (+ 1754us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:36.030287 (+ 1644us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:36.030295 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:36.030298 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:36.030564 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:36.030827 (+ 263us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:36.030828 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:36.030828 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:36.030940 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:36.031226 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:36.031232 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:36.031375 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 15:00:36.031503 (+ 128us) client_negotiation.cc:240] Negotiation successful
0926 15:00:36.031669 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":81,"threads_started":1}
I20260926 15:00:36.032447 14635 heartbeater.cc:347] Connected to a master server at 127.8.102.62:44261
I20260926 15:00:36.032572 14635 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:36.032835 14635 heartbeater.cc:511] Master 127.8.102.62:44261 requested a full tablet report, sending...
I20260926 15:00:36.033267 14176 ts_manager.cc:194] Registered new tserver with Master: 4a04fcf168034df2b21ceed31cd04081 (127.8.102.3:45101)
I20260926 15:00:36.033656 14176 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:37297
I20260926 15:00:36.040086 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/test-admin.keytab.
Sep 26 15:00:36 dist-test-slave-jnhs krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:36.056250 8600 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260926 15:00:36.057873 14642 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:36.057101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33496 (local address 127.8.102.1:46735)
0926 15:00:36.057273 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:36.057277 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:36.057286 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:36.057311 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:36.057313 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:36.057327 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:36.057407 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:36.057787 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33496: BlockingRecv error: recv got EOF from 127.0.0.1:33496 (error 108)
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":72,"threads_started":1}
W20260926 15:00:36.058005 8600 init.cc:244] Error while opening credential cache '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/krb5cc)
Sep 26 15:00:36 dist-test-slave-jnhs krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:36.059005 8600 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 15:00:36 dist-test-slave-jnhs krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:36.059875 8600 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 15:00:36 dist-test-slave-jnhs krb5kdc[14131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.1@KRBTEST.COM
I20260926 15:00:36.067947 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:36.060397 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33506 (local address 127.8.102.1:46735)
0926 15:00:36.060460 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:36.060463 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:36.060526 (+ 63us) server_negotiation.cc:407] Connection header received
0926 15:00:36.060562 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:36.060563 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:36.060574 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:36.060640 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:36.061062 (+ 422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:36.061795 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:36.062553 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:36.062963 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:36.063048 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:36.065156 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:36.065166 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:36.065167 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:36.065174 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:36.066730 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:36.067098 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:36.067103 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:36.067103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:36.067165 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:36.067377 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:36.067378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:36.067379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:36.067528 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:36.067694 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:36.067742 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 15:00:36.067822 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20260926 15:00:36.069784 14642 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:36.069202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33510 (local address 127.8.102.1:46735)
0926 15:00:36.069256 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:36.069258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:36.069324 (+ 66us) server_negotiation.cc:407] Connection header received
0926 15:00:36.069386 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:36.069387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:36.069398 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:36.069452 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:36.069734 (+ 282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33510: BlockingRecv error: recv got EOF from 127.0.0.1:33510 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
W20260926 15:00:36.069864 8600 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/krb5cc' of type FILE
Sep 26 15:00:36 dist-test-slave-jnhs krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:36.071009 8600 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 15:00:36 dist-test-slave-jnhs krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:36.071921 8600 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 15:00:36 dist-test-slave-jnhs krb5kdc[14131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.1@KRBTEST.COM
I20260926 15:00:36.079304 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:36.072582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33522 (local address 127.8.102.1:46735)
0926 15:00:36.072622 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:36.072624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:36.072666 (+ 42us) server_negotiation.cc:407] Connection header received
0926 15:00:36.072743 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:36.072744 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:36.072754 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:36.072843 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:36.073254 (+ 411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:36.073779 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:36.074474 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:36.074759 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:36.074844 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:36.076864 (+ 2020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:36.076866 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:36.076866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:36.076874 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:36.078094 (+ 1220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:36.078521 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:36.078523 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:36.078523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:36.078572 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:36.078801 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:36.078802 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:36.078802 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:36.078908 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:36.079011 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:36.079129 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 15:00:36.079207 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 15:00:36.080782 14642 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:36.080321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33536 (local address 127.8.102.1:46735)
0926 15:00:36.080358 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:36.080360 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:36.080371 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:00:36.080419 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:36.080419 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:36.080426 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:36.080506 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:36.080740 (+ 234us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33536: BlockingRecv error: recv got EOF from 127.0.0.1:33536 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 15:00:36.080894 8600 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790434810362722-8600-0/krb5kdc/krb5cc' of type FILE
Sep 26 15:00:36 dist-test-slave-jnhs krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:36.081962 8600 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 26 15:00:36 dist-test-slave-jnhs krb5kdc[14131](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:36.082991 8600 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 26 15:00:36 dist-test-slave-jnhs krb5kdc[14131](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434836, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.1@KRBTEST.COM
I20260926 15:00:36.090077 14642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:36.083557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33548 (local address 127.8.102.1:46735)
0926 15:00:36.083619 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:36.083622 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:36.083686 (+ 64us) server_negotiation.cc:407] Connection header received
0926 15:00:36.083763 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:36.083764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:36.083772 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:36.083838 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:36.084227 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:36.084693 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:36.085370 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:36.085564 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:36.085639 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:36.087764 (+ 2125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:36.087765 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:36.087765 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:36.087771 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:36.089010 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:36.089377 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:36.089378 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:36.089379 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:36.089433 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:36.089629 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:36.089630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:36.089630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:36.089727 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:36.089800 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:36.089939 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 15:00:36.089999 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 15:00:36.090855 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 14238
I20260926 15:00:36.095583 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 14372
I20260926 15:00:36.099804 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 14506
I20260926 15:00:36.104060 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 14144
2026-09-26T15:00:36Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1899 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:00:36 dist-test-slave-jnhs krb5kdc[14655](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:00:36 dist-test-slave-jnhs krb5kdc[14655](info): set up 2 sockets
Sep 26 15:00:36 dist-test-slave-jnhs krb5kdc[14655](info): commencing operation
krb5kdc: starting...
W20260926 15:00:37.151384 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.027s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:00:37 dist-test-slave-jnhs krb5kdc[14655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:00:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:00:37Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.62.keytab.
Entry for principal oryx/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.62.keytab.
I20260926 15:00:37.279211 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:44775
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.62.keytab
--principal=oryx/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:44775
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:37.362075 14668 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:37.362299 14668 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:37.362322 14668 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:37.363443 14668 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:37.363483 14668 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:37.363497 14668 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:37.363510 14668 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:37.363523 14668 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:00:37.365031 14668 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:44775
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.62.keytab
--principal=oryx/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:44775
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14668
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:37.365303 14668 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:37.365545 14668 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:37.370370 14676 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:37.370396 14674 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:37.370393 14673 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:37.370760 14668 server_base.cc:1034] running on GCE node
I20260926 15:00:37.371059 14668 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:37.371384 14668 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:37.372587 14668 hybrid_clock.cc:648] HybridClock initialized: now 1790434837372556 us; error 39 us; skew 500 ppm
Sep 26 15:00:37 dist-test-slave-jnhs krb5kdc[14655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434837, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:37.382797 14668 init.cc:377] Logged in from keytab as oryx/127.8.102.62@KRBTEST.COM (short username oryx)
I20260926 15:00:37.383347 14668 webserver.cc:460] Webserver started at http://127.8.102.62:33077/ using document root <none> and password file <none>
I20260926 15:00:37.383625 14668 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:37.383708 14668 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:37.383895 14668 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:37.385102 14668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "353bc184be504361bc9d470653da6fdc"
format_stamp: "Formatted at 2026-09-26 15:00:37 on dist-test-slave-jnhs"
server_key: "10e5704353712b05d28a0906f3ace70f"
server_key_iv: "433dd662c23f1e9e0e1f9d64d5a28b4b"
server_key_version: "encryptionkey@0"
I20260926 15:00:37.385644 14668 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "353bc184be504361bc9d470653da6fdc"
format_stamp: "Formatted at 2026-09-26 15:00:37 on dist-test-slave-jnhs"
server_key: "10e5704353712b05d28a0906f3ace70f"
server_key_iv: "433dd662c23f1e9e0e1f9d64d5a28b4b"
server_key_version: "encryptionkey@0"
I20260926 15:00:37.387260 14668 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:37.388229 14683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:37.388425 14668 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:37.388597 14668 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "353bc184be504361bc9d470653da6fdc"
format_stamp: "Formatted at 2026-09-26 15:00:37 on dist-test-slave-jnhs"
server_key: "10e5704353712b05d28a0906f3ace70f"
server_key_iv: "433dd662c23f1e9e0e1f9d64d5a28b4b"
server_key_version: "encryptionkey@0"
I20260926 15:00:37.388687 14668 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:37.429852 14668 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:37.432626 14668 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:37.432802 14668 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:37.436690 14668 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:44775
I20260926 15:00:37.436743 14735 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:44775 every 8 connection(s)
I20260926 15:00:37.437125 14668 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:00:37.438158 14736 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:37.440541 14736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc: Bootstrap starting.
I20260926 15:00:37.441115 14736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:37.441360 14736 log.cc:824] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:37.442070 14736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc: No bootstrap required, opened a new log
I20260926 15:00:37.443665 14736 raft_consensus.cc:348] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } }
I20260926 15:00:37.443813 14736 raft_consensus.cc:374] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:37.443837 14736 raft_consensus.cc:729] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 353bc184be504361bc9d470653da6fdc, State: Initialized, Role: FOLLOWER
I20260926 15:00:37.443984 14736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } }
I20260926 15:00:37.444068 14736 raft_consensus.cc:388] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:00:37.444092 14736 raft_consensus.cc:482] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:00:37.444123 14736 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:37.445000 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 14668
I20260926 15:00:37.445050 14736 raft_consensus.cc:504] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } }
I20260926 15:00:37.445120 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:00:37.445158 14736 leader_election.cc:302] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 353bc184be504361bc9d470653da6fdc; no voters:
I20260926 15:00:37.445331 8600 external_mini_cluster.cc:1428] Setting key 3acf5a69795b012ff8a0232cd986cd25
I20260926 15:00:37.445345 14736 leader_election.cc:288] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:00:37.445406 14739 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:37.445575 14739 raft_consensus.cc:686] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [term 1 LEADER]: Becoming Leader. State: Replica: 353bc184be504361bc9d470653da6fdc, State: Running, Role: LEADER
I20260926 15:00:37.445611 14736 sys_catalog.cc:564] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:37.445677 14739 consensus_queue.cc:237] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } }
I20260926 15:00:37.446152 14740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "353bc184be504361bc9d470653da6fdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } }
I20260926 15:00:37.446254 14740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:37.446412 14741 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 353bc184be504361bc9d470653da6fdc. Latest consensus state: current_term: 1 leader_uuid: "353bc184be504361bc9d470653da6fdc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } }
I20260926 15:00:37.446470 14741 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:37.446467 14750 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:00:37.447395 14750 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:00:37.449174 14750 catalog_manager.cc:1347] Generated new cluster ID: 509e0d816e344dd3ab0df33248b2f26b
I20260926 15:00:37.449239 14750 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 15:00:37 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM
I20260926 15:00:37.457695 14750 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:00:37.458379 14750 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:00:37.459278 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:37.446020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37214 (local address 127.8.102.62:44775)
0926 15:00:37.446284 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:37.446295 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:37.446377 (+ 82us) server_negotiation.cc:407] Connection header received
0926 15:00:37.446818 (+ 441us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:37.446828 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:37.446896 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:37.447105 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:37.448067 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:37.449308 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:37.450440 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:37.451123 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:37.451228 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:37.454246 (+ 3018us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:37.454252 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:37.454252 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:37.454270 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:37.457317 (+ 3047us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:37.457778 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:37.457782 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:37.457782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:37.457890 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:37.458133 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:37.458133 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:37.458134 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:37.458418 (+ 284us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:37.458581 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:37.458861 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 15:00:37.459041 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":62,"threads_started":1}
I20260926 15:00:37.460829 14750 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc: Generated new TSK 0
I20260926 15:00:37.461023 14750 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.1.keytab.
Entry for principal oryx/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.1.keytab.
I20260926 15:00:37.491235 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44775
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.1.keytab
--principal=oryx/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:37.574061 14762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:37.574308 14762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:37.574352 14762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:37.575625 14762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:37.575697 14762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:37.575745 14762 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:00:37.577286 14762 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.1.keytab
--principal=oryx/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14762
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:37.577564 14762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:37.577796 14762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:37.583056 14770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:37.583056 14767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:37.583091 14768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:37.583114 14762 server_base.cc:1034] running on GCE node
I20260926 15:00:37.583468 14762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:37.583801 14762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:37.584969 14762 hybrid_clock.cc:648] HybridClock initialized: now 1790434837584947 us; error 39 us; skew 500 ppm
Sep 26 15:00:37 dist-test-slave-jnhs krb5kdc[14655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434837, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:37.587930 14762 init.cc:377] Logged in from keytab as oryx/127.8.102.1@KRBTEST.COM (short username oryx)
I20260926 15:00:37.588470 14762 webserver.cc:460] Webserver started at http://127.8.102.1:41323/ using document root <none> and password file <none>
I20260926 15:00:37.588665 14762 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:37.588706 14762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:37.588796 14762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:37.589836 14762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "9425f4a721a6474bb40adb111db99615"
format_stamp: "Formatted at 2026-09-26 15:00:37 on dist-test-slave-jnhs"
server_key: "944691509257e51fdd43de592e951a54"
server_key_iv: "8e0a1db909e68d8b0d34c07e0c21353f"
server_key_version: "encryptionkey@0"
I20260926 15:00:37.590140 14762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "9425f4a721a6474bb40adb111db99615"
format_stamp: "Formatted at 2026-09-26 15:00:37 on dist-test-slave-jnhs"
server_key: "944691509257e51fdd43de592e951a54"
server_key_iv: "8e0a1db909e68d8b0d34c07e0c21353f"
server_key_version: "encryptionkey@0"
I20260926 15:00:37.591583 14762 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 15:00:37.592402 14777 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:37.592597 14762 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 15:00:37.592685 14762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "9425f4a721a6474bb40adb111db99615"
format_stamp: "Formatted at 2026-09-26 15:00:37 on dist-test-slave-jnhs"
server_key: "944691509257e51fdd43de592e951a54"
server_key_iv: "8e0a1db909e68d8b0d34c07e0c21353f"
server_key_version: "encryptionkey@0"
I20260926 15:00:37.592744 14762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:37.630375 14762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:37.632778 14762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:37.632896 14762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:37.633126 14762 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:37.633446 14762 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:37.633481 14762 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:37.633523 14762 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:37.633550 14762 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:37.639380 14762 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:35343
I20260926 15:00:37.639499 14890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:35343 every 8 connection(s)
I20260926 15:00:37.639794 14762 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
Sep 26 15:00:37 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434837, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM
I20260926 15:00:37.646984 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 14762
I20260926 15:00:37.647102 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:00:37.647337 8600 external_mini_cluster.cc:1428] Setting key be6cbb7ab87dcf35f769f47304bf307e
I20260926 15:00:37.649525 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:37.640569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:47491 (local address 127.8.102.62:44775)
0926 15:00:37.640651 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:37.640654 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:37.640799 (+ 145us) server_negotiation.cc:407] Connection header received
0926 15:00:37.641282 (+ 483us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:37.641283 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:37.641294 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:37.641349 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:37.642314 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:37.642886 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:37.643861 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:37.644056 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:37.644114 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:37.646269 (+ 2155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:37.646270 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:37.646271 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:37.646277 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:37.648133 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:37.648549 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:37.648550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:37.648550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:37.648615 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:37.648862 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:37.648863 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:37.648863 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:37.648988 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:37.649064 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:37.649344 (+ 280us) server_negotiation.cc:299] Negotiation successful
0926 15:00:37.649435 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260926 15:00:37.649734 14893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:37.640510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.1:47491)
0926 15:00:37.640682 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:37.640720 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:37.641197 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:37.641405 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:37.641408 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:37.642009 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:37.642207 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:37.642211 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:37.642969 (+ 758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:37.642970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:37.643788 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:37.643790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:37.644138 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:37.644139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:37.644242 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:37.644715 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:37.644724 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:37.646172 (+ 1448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:37.648216 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:37.648223 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:37.648226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:37.648470 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:37.648681 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:37.648681 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:37.648682 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:37.648795 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:37.649056 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:37.649062 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:37.649207 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 15:00:37.649311 (+ 104us) client_negotiation.cc:240] Negotiation successful
0926 15:00:37.649484 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":89,"threads_started":1}
I20260926 15:00:37.650202 14891 heartbeater.cc:347] Connected to a master server at 127.8.102.62:44775
I20260926 15:00:37.650296 14891 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:37.650439 14891 heartbeater.cc:511] Master 127.8.102.62:44775 requested a full tablet report, sending...
I20260926 15:00:37.651006 14700 ts_manager.cc:194] Registered new tserver with Master: 9425f4a721a6474bb40adb111db99615 (127.8.102.1:35343)
I20260926 15:00:37.651728 14700 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:47491
WARNING: no policy specified for oryx/127.8.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.2.keytab.
Entry for principal oryx/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.2.keytab.
I20260926 15:00:37.677533 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44775
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.2.keytab
--principal=oryx/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:37.759958 14896 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:37.760169 14896 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:37.760192 14896 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:37.761387 14896 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:37.761443 14896 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:37.761471 14896 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:00:37.762990 14896 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.2.keytab
--principal=oryx/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.14896
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:37.763258 14896 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:37.763556 14896 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:37.768882 14901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:37.768909 14902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:37.768903 14904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:37.769196 14896 server_base.cc:1034] running on GCE node
I20260926 15:00:37.769394 14896 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:37.769606 14896 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:37.770711 14896 hybrid_clock.cc:648] HybridClock initialized: now 1790434837770676 us; error 43 us; skew 500 ppm
Sep 26 15:00:37 dist-test-slave-jnhs krb5kdc[14655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434837, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:37.773175 14896 init.cc:377] Logged in from keytab as oryx/127.8.102.2@KRBTEST.COM (short username oryx)
I20260926 15:00:37.773634 14896 webserver.cc:460] Webserver started at http://127.8.102.2:36803/ using document root <none> and password file <none>
I20260926 15:00:37.773821 14896 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:37.773864 14896 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:37.773953 14896 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:37.775046 14896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "5d53cf9c2da74a33bb309bdea1008cd5"
format_stamp: "Formatted at 2026-09-26 15:00:37 on dist-test-slave-jnhs"
server_key: "784191f64e1ee5873a10b8ff521fe825"
server_key_iv: "7e444ae663f9632d7eaaad0a4a00a133"
server_key_version: "encryptionkey@0"
I20260926 15:00:37.775346 14896 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "5d53cf9c2da74a33bb309bdea1008cd5"
format_stamp: "Formatted at 2026-09-26 15:00:37 on dist-test-slave-jnhs"
server_key: "784191f64e1ee5873a10b8ff521fe825"
server_key_iv: "7e444ae663f9632d7eaaad0a4a00a133"
server_key_version: "encryptionkey@0"
I20260926 15:00:37.776647 14896 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:00:37.777446 14911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:37.777756 14896 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:37.777873 14896 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "5d53cf9c2da74a33bb309bdea1008cd5"
format_stamp: "Formatted at 2026-09-26 15:00:37 on dist-test-slave-jnhs"
server_key: "784191f64e1ee5873a10b8ff521fe825"
server_key_iv: "7e444ae663f9632d7eaaad0a4a00a133"
server_key_version: "encryptionkey@0"
I20260926 15:00:37.777954 14896 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:37.786923 14896 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:37.789021 14896 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:37.789201 14896 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:37.789446 14896 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:37.789754 14896 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:37.789808 14896 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:37.789862 14896 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:37.789897 14896 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:37.796108 14896 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:33887
I20260926 15:00:37.796169 15024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:33887 every 8 connection(s)
I20260926 15:00:37.796517 14896 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
Sep 26 15:00:37 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434837, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM
I20260926 15:00:37.802837 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 14896
I20260926 15:00:37.802963 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:00:37.803218 8600 external_mini_cluster.cc:1428] Setting key 526bbbdc6434cfad103a92d57835c20f
I20260926 15:00:37.806183 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:37.797283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:54575 (local address 127.8.102.62:44775)
0926 15:00:37.797352 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:37.797354 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:37.797566 (+ 212us) server_negotiation.cc:407] Connection header received
0926 15:00:37.798082 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:37.798084 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:37.798096 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:37.798155 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:37.799153 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:37.799650 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:37.800651 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:37.800864 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:37.800949 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:37.803053 (+ 2104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:37.803055 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:37.803055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:37.803065 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:37.804859 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:37.805254 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:37.805255 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:37.805256 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:37.805321 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:37.805557 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:37.805559 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:37.805559 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:37.805683 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:37.805756 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:37.806009 (+ 253us) server_negotiation.cc:299] Negotiation successful
0926 15:00:37.806088 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 15:00:37.806421 15027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:37.797262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.2:54575)
0926 15:00:37.797443 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:37.797477 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:37.797981 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:37.798229 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:37.798232 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:37.798841 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:37.799039 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:37.799043 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:37.799742 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:37.799743 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:37.800555 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:37.800557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:37.800965 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:37.800966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:37.801083 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:37.801549 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:37.801557 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:37.802918 (+ 1361us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:37.804962 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:37.804968 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:37.804970 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:37.805186 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:37.805392 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:37.805393 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:37.805393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:37.805486 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:37.805747 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:37.805753 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:37.805881 (+ 128us) client_negotiation.cc:769] Sending connection context
0926 15:00:37.805988 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 15:00:37.806152 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":86,"threads_started":1}
I20260926 15:00:37.807014 15025 heartbeater.cc:347] Connected to a master server at 127.8.102.62:44775
I20260926 15:00:37.807117 15025 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:37.807288 15025 heartbeater.cc:511] Master 127.8.102.62:44775 requested a full tablet report, sending...
I20260926 15:00:37.807684 14700 ts_manager.cc:194] Registered new tserver with Master: 5d53cf9c2da74a33bb309bdea1008cd5 (127.8.102.2:33887)
I20260926 15:00:37.808063 14700 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:54575
WARNING: no policy specified for oryx/127.8.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.3.keytab.
Entry for principal oryx/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.3.keytab.
I20260926 15:00:37.830576 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44775
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.3.keytab
--principal=oryx/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:37.913721 15030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:37.913941 15030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:37.913966 15030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:37.915146 15030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:37.915203 15030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:37.915247 15030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:00:37.916734 15030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.3.keytab
--principal=oryx/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15030
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:37.916951 15030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:37.917146 15030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 15:00:37.922856 15030 server_base.cc:1034] running on GCE node
W20260926 15:00:37.922822 15036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:37.922822 15035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:37.922860 15038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:37.923295 15030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:37.923570 15030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:37.924726 15030 hybrid_clock.cc:648] HybridClock initialized: now 1790434837924699 us; error 44 us; skew 500 ppm
Sep 26 15:00:37 dist-test-slave-jnhs krb5kdc[14655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434837, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:37.927412 15030 init.cc:377] Logged in from keytab as oryx/127.8.102.3@KRBTEST.COM (short username oryx)
I20260926 15:00:37.927889 15030 webserver.cc:460] Webserver started at http://127.8.102.3:32847/ using document root <none> and password file <none>
I20260926 15:00:37.928139 15030 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:37.928186 15030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:37.928365 15030 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:37.929450 15030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "e82ec9077f6244fa83a17bfcfe8af69a"
format_stamp: "Formatted at 2026-09-26 15:00:37 on dist-test-slave-jnhs"
server_key: "b297f0fdfe24eb5109161d9149d325bc"
server_key_iv: "60c22a5a8fd9e04b615631e89a2bb433"
server_key_version: "encryptionkey@0"
I20260926 15:00:37.929807 15030 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "e82ec9077f6244fa83a17bfcfe8af69a"
format_stamp: "Formatted at 2026-09-26 15:00:37 on dist-test-slave-jnhs"
server_key: "b297f0fdfe24eb5109161d9149d325bc"
server_key_iv: "60c22a5a8fd9e04b615631e89a2bb433"
server_key_version: "encryptionkey@0"
I20260926 15:00:37.931264 15030 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 15:00:37.932101 15045 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:37.932305 15030 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:37.932410 15030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "e82ec9077f6244fa83a17bfcfe8af69a"
format_stamp: "Formatted at 2026-09-26 15:00:37 on dist-test-slave-jnhs"
server_key: "b297f0fdfe24eb5109161d9149d325bc"
server_key_iv: "60c22a5a8fd9e04b615631e89a2bb433"
server_key_version: "encryptionkey@0"
I20260926 15:00:37.932484 15030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:37.946002 15030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:37.948431 15030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:37.948602 15030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:37.948829 15030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:37.949133 15030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:37.949185 15030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:37.949227 15030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:37.949262 15030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:37.955183 15030 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:36979
I20260926 15:00:37.955228 15158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:36979 every 8 connection(s)
I20260926 15:00:37.955551 15030 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:00:37.955961 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 15030
I20260926 15:00:37.956082 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:00:37.956303 8600 external_mini_cluster.cc:1428] Setting key 98bddad7d40ec17b233c37bb63f90f96
Sep 26 15:00:37 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434837, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM
I20260926 15:00:37.965317 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:37.956309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:59029 (local address 127.8.102.62:44775)
0926 15:00:37.956387 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:37.956389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:37.956664 (+ 275us) server_negotiation.cc:407] Connection header received
0926 15:00:37.957298 (+ 634us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:37.957299 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:37.957310 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:37.957361 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:37.958271 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:37.958850 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:37.959851 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:37.960051 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:37.960119 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:37.962485 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:37.962487 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:37.962487 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:37.962495 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:37.963805 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:37.964214 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:37.964215 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:37.964215 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:37.964265 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:37.964509 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:37.964509 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:37.964510 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:37.964616 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:37.964667 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:37.965129 (+ 462us) server_negotiation.cc:299] Negotiation successful
0926 15:00:37.965221 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 15:00:37.965544 15161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:37.956306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.3:59029)
0926 15:00:37.956488 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:37.956570 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:37.957197 (+ 627us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:37.957423 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:37.957427 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:37.957992 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:37.958182 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:37.958186 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:37.958931 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:37.958932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:37.959751 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:37.959754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:37.960159 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:37.960160 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:37.960281 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:37.960953 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:37.960962 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:37.962383 (+ 1421us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:37.963881 (+ 1498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:37.963887 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:37.963889 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:37.964149 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:37.964343 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:37.964343 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:37.964344 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:37.964454 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:37.964668 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:37.964672 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:37.964963 (+ 291us) client_negotiation.cc:769] Sending connection context
0926 15:00:37.965080 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 15:00:37.965233 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":112,"threads_started":1}
I20260926 15:00:37.966127 15159 heartbeater.cc:347] Connected to a master server at 127.8.102.62:44775
I20260926 15:00:37.966220 15159 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:37.966363 15159 heartbeater.cc:511] Master 127.8.102.62:44775 requested a full tablet report, sending...
I20260926 15:00:37.966778 14700 ts_manager.cc:194] Registered new tserver with Master: e82ec9077f6244fa83a17bfcfe8af69a (127.8.102.3:36979)
I20260926 15:00:37.967226 14700 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:59029
I20260926 15:00:37.968045 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.102.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.102.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.102.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.61.keytab.
Entry for principal oryx/127.8.102.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.61.keytab.
I20260926 15:00:37.995788 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.61:34051
--webserver_interface=127.8.102.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.61.keytab
--principal=oryx/127.8.102.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:44775,127.8.102.61:34051
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:38.078750 15164 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:38.078979 15164 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:38.079003 15164 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:38.080284 15164 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:38.080336 15164 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:38.080351 15164 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:38.080364 15164 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:38.080377 15164 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:00:38.082042 15164 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:44775,127.8.102.61:34051
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.61.keytab
--principal=oryx/127.8.102.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.61:34051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.102.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15164
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
W20260926 15:00:38.082170 15164 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.8.102.62:44775,127.8.102.61:34051'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260926 15:00:38.082355 15164 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:38.082599 15164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:38.087549 15170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:38.087584 15169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:38.087620 15164 server_base.cc:1034] running on GCE node
W20260926 15:00:38.087585 15172 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:38.088076 15164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:38.088341 15164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:38.089511 15164 hybrid_clock.cc:648] HybridClock initialized: now 1790434838089493 us; error 32 us; skew 500 ppm
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:38.092449 15164 init.cc:377] Logged in from keytab as oryx/127.8.102.61@KRBTEST.COM (short username oryx)
I20260926 15:00:38.092981 15164 webserver.cc:460] Webserver started at http://127.8.102.61:37123/ using document root <none> and password file <none>
I20260926 15:00:38.093222 15164 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:38.093287 15164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:38.093422 15164 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:38.094496 15164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/data/instance:
uuid: "26a22fd6ed974376bb5a6d4da9a14957"
format_stamp: "Formatted at 2026-09-26 15:00:38 on dist-test-slave-jnhs"
server_key: "1303575623b346032b5941b2780ad938"
server_key_iv: "9613bde1d6c20cf9e342ea442bbec29a"
server_key_version: "encryptionkey@0"
I20260926 15:00:38.094898 15164 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/wal/instance:
uuid: "26a22fd6ed974376bb5a6d4da9a14957"
format_stamp: "Formatted at 2026-09-26 15:00:38 on dist-test-slave-jnhs"
server_key: "1303575623b346032b5941b2780ad938"
server_key_iv: "9613bde1d6c20cf9e342ea442bbec29a"
server_key_version: "encryptionkey@0"
I20260926 15:00:38.096443 15164 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:38.097193 15179 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:38.097397 15164 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:38.097507 15164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/wal
uuid: "26a22fd6ed974376bb5a6d4da9a14957"
format_stamp: "Formatted at 2026-09-26 15:00:38 on dist-test-slave-jnhs"
server_key: "1303575623b346032b5941b2780ad938"
server_key_iv: "9613bde1d6c20cf9e342ea442bbec29a"
server_key_version: "encryptionkey@0"
I20260926 15:00:38.097595 15164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:38.120141 15164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:38.122452 15164 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:38.122659 15164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:38.126590 15164 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.61:34051
I20260926 15:00:38.126674 15231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.61:34051 every 8 connection(s)
I20260926 15:00:38.127041 15164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/data/info.pb
I20260926 15:00:38.128041 15232 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:38.129657 15232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:00:38.131361 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 15164
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.61@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.102.61@KRBTEST.COM
I20260926 15:00:38.140887 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.130393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37222 (local address 127.8.102.62:44775)
0926 15:00:38.130454 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.130457 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.130689 (+ 232us) server_negotiation.cc:407] Connection header received
0926 15:00:38.131217 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.131218 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.131232 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.131318 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.132314 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.133136 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.134691 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.135005 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.135082 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.137426 (+ 2344us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.137427 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.137428 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.137439 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.139032 (+ 1593us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.139714 (+ 682us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.139715 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.139715 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.139771 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.140098 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.140099 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.140100 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.140263 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.140375 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.140696 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.140782 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 15:00:38.141101 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.130353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.0.0.1:37222)
0926 15:00:38.130544 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.130586 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.131090 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.131493 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.131498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.131879 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.132183 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.132190 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.133233 (+ 1043us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.133234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.134581 (+ 1347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.134583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.135106 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.135106 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.135215 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.135732 (+ 517us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.135743 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.137323 (+ 1580us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.139148 (+ 1825us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.139155 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.139157 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.139620 (+ 463us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.139832 (+ 212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.139833 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.139834 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.140003 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.140363 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.140370 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.140549 (+ 179us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.140674 (+ 125us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.140834 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":5,"thread_start_us":109,"threads_started":1}
I20260926 15:00:38.141611 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.131951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49792 (local address 127.8.102.61:34051)
0926 15:00:38.132102 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.132107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.132145 (+ 38us) server_negotiation.cc:407] Connection header received
0926 15:00:38.132210 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.132213 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.132234 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.132305 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.133074 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.134496 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.135517 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.136007 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.136082 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.138349 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.138357 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.138357 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.138363 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.140201 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.140560 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.140563 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.140564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.140624 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.140820 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.140821 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.140822 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.141007 (+ 185us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.141166 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.141385 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.141477 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":92,"threads_started":1}
I20260926 15:00:38.141772 15232 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.61@KRBTEST.COM for oryx/127.8.102.61@KRBTEST.COM
I20260926 15:00:38.148414 15234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.142031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:34051 (local address 127.0.0.1:49798)
0926 15:00:38.142081 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.142091 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.142132 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.142257 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.142258 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.142391 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.142450 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.142451 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.143222 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.143223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.143706 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.143708 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.144072 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.144073 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.144134 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.144520 (+ 386us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.144524 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.145783 (+ 1259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.147466 (+ 1683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.147467 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.147467 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.147652 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.147839 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.147840 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.147840 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.147900 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.148136 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.148137 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.148177 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.148245 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.148328 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 15:00:38.148495 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.142094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49798 (local address 127.8.102.61:34051)
0926 15:00:38.142127 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.142129 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.142137 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:00:38.142197 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.142198 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.142209 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.142256 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.142563 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.143129 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.143797 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.144007 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.144077 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.145873 (+ 1796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.145874 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.145874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.145880 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.147354 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.147716 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.147717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.147717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.147763 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.147961 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.147962 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.147962 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.148058 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.148107 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.148270 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.148372 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":42,"server-negotiator.queue_time_us":17}
I20260926 15:00:38.149796 15232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: Bootstrap starting.
I20260926 15:00:38.150743 15232 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:38.151034 15232 log.cc:824] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:38.151736 15232 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: No bootstrap required, opened a new log
I20260926 15:00:38.153152 15232 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } }
I20260926 15:00:38.153301 15232 raft_consensus.cc:374] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:38.153347 15232 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26a22fd6ed974376bb5a6d4da9a14957, State: Initialized, Role: FOLLOWER
I20260926 15:00:38.153497 15232 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } }
I20260926 15:00:38.153749 15238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } } }
I20260926 15:00:38.153806 15232 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:38.153838 15238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 15:00:38.155668 15249 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:00:38.155743 15249 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:00:38.157330 15164 master_runner.cc:428] Detected that this master 26a22fd6ed974376bb5a6d4da9a14957 is joining an existing cluster
I20260926 15:00:38.157387 15164 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.102.61:34051
I20260926 15:00:38.165133 15256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.158589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:34051 (local address 127.0.0.1:49800)
0926 15:00:38.159006 (+ 417us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.159016 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.159086 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.159212 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.159212 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.159399 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.159480 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.159482 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.160307 (+ 825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.160308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.161167 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.161169 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.161594 (+ 425us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.161595 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.161668 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.162025 (+ 357us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.162030 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.162336 (+ 306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.164007 (+ 1671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.164008 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.164008 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.164221 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.164420 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.164420 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.164420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.164483 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.164742 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.164743 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.164779 (+ 36us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.164882 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.164992 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":82,"threads_started":1}
I20260926 15:00:38.165261 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.158440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37226 (local address 127.8.102.62:44775)
0926 15:00:38.158488 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.158491 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.158588 (+ 97us) server_negotiation.cc:407] Connection header received
0926 15:00:38.158733 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.158734 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.158745 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.158775 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.159206 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.159859 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.160726 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.160937 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.161047 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.162184 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.162186 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.162186 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.162195 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.163938 (+ 1743us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.164401 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.164402 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.164403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.164461 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.164693 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.164695 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.164695 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.164833 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.164878 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.165082 (+ 204us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.165163 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 15:00:38.165140 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.158624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49800 (local address 127.8.102.61:34051)
0926 15:00:38.158709 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.158711 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.159095 (+ 384us) server_negotiation.cc:407] Connection header received
0926 15:00:38.159140 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.159140 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.159151 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.159207 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.159561 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.160223 (+ 662us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.161251 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.161468 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.161552 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.162410 (+ 858us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.162411 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.162411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.162417 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.163907 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.164301 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.164302 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.164302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.164352 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.164560 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.164561 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.164561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.164670 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.164739 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.164903 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.165054 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":68}
I20260926 15:00:38.165550 15255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.158371 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.0.0.1:37226)
0926 15:00:38.158522 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.158533 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.158616 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.158837 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.158838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.159000 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.159108 (+ 108us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.159110 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.159949 (+ 839us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.159950 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.160644 (+ 694us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.160646 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.161060 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.161061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.161128 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.161650 (+ 522us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.161657 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.162089 (+ 432us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.164036 (+ 1947us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.164037 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.164038 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.164318 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.164529 (+ 211us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.164529 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.164530 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.164604 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.164926 (+ 322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.164927 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.164969 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.165336 (+ 367us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.165446 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":86,"threads_started":1}
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434837, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.62@KRBTEST.COM for oryx/127.8.102.61@KRBTEST.COM
I20260926 15:00:38.175609 15236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.167964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49802 (local address 127.8.102.61:34051)
0926 15:00:38.168003 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.168005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.168172 (+ 167us) server_negotiation.cc:407] Connection header received
0926 15:00:38.168262 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.168262 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.168270 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.168316 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.168844 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.169296 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.170456 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.170702 (+ 246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.170770 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.172947 (+ 2177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.172949 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.172949 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.172956 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.174347 (+ 1391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.174773 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.174775 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.174775 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.174832 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.175039 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.175040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.175040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.175152 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.175225 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.175402 (+ 177us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.175507 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 15:00:38.175625 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.167912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:34051 (local address 127.0.0.1:49802)
0926 15:00:38.168063 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.168104 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.168186 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.168321 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.168324 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.168682 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.168745 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.168747 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.169376 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.169377 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.170324 (+ 947us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.170326 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.170790 (+ 464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.170792 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.170882 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.171319 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.171325 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.172843 (+ 1518us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.174424 (+ 1581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.174433 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.174435 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.174708 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.174897 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.174898 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.174899 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.174960 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.175218 (+ 258us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.175225 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.175281 (+ 56us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.175377 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.175506 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
I20260926 15:00:38.176319 14700 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.102.61:34051: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957"
member_type: NON_VOTER
last_known_addr {
host: "127.8.102.61"
port: 34051
}
attrs {
promote: true
}
}
dest_uuid: "353bc184be504361bc9d470653da6fdc"
cas_config_opid_index: -1
I20260926 15:00:38.176679 14700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: NON_VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: true } }
I20260926 15:00:38.177227 15259 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 26a22fd6ed974376bb5a6d4da9a14957 (127.8.102.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: NON_VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: true } } }
I20260926 15:00:38.177525 15206 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:38.178304 15259 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.102.61:34051
I20260926 15:00:38.178318 15206 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 1 FOLLOWER]: Refusing update from remote peer 353bc184be504361bc9d470653da6fdc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260926 15:00:38.178454 15258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "353bc184be504361bc9d470653da6fdc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: NON_VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: true } } }
I20260926 15:00:38.178666 15164 master.cc:537] Master@127.8.102.61:34051 shutting down...
I20260926 15:00:38.178679 15258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:38.178660 15259 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: NON_VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260926 15:00:38.179378 14687 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260926 15:00:38.181478 15164 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:38.181588 15164 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:38.181651 15164 tablet_replica.cc:330] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: stopping tablet replica
I20260926 15:00:38.187594 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.187446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:34051 (local address 127.0.0.1:49806)
0926 15:00:38.187509 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.187560 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.61:34051: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 15:00:38.187726 14687 consensus_peers.cc:487] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc -> Peer 26a22fd6ed974376bb5a6d4da9a14957 (127.8.102.61:34051): Couldn't send request to peer 26a22fd6ed974376bb5a6d4da9a14957. Status: Network error: Client connection negotiation failed: client connection to 127.8.102.61:34051: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 15:00:38.194332 15164 master.cc:552] Master@127.8.102.61:34051 shutdown complete.
I20260926 15:00:38.194391 15164 master_runner.cc:305] Clearing existing system tablet
I20260926 15:00:38.194540 15164 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:38.194592 15164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:38.195393 15164 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:38.196053 15263 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:38.196183 15164 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:38.196241 15164 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:38.196269 15164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/wal
uuid: "26a22fd6ed974376bb5a6d4da9a14957"
format_stamp: "Formatted at 2026-09-26 15:00:38 on dist-test-slave-jnhs"
server_key: "1303575623b346032b5941b2780ad938"
server_key_iv: "9613bde1d6c20cf9e342ea442bbec29a"
server_key_version: "encryptionkey@0"
I20260926 15:00:38.197116 15164 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:38.198861 15164 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 15:00:38.198956 15164 log.cc:1196] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260926 15:00:38.199193 15164 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: Deleting consensus metadata
I20260926 15:00:38.199326 15164 master_runner.cc:315] Copying system tablet from 127.8.102.62:44775
I20260926 15:00:38.200032 15164 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: tablet copy: Beginning tablet copy session from remote peer at address 127.8.102.62:44775
I20260926 15:00:38.206740 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.200445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37230 (local address 127.8.102.62:44775)
0926 15:00:38.200512 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.200515 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.200658 (+ 143us) server_negotiation.cc:407] Connection header received
0926 15:00:38.200696 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.200697 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.200710 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.200769 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.201096 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.201893 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.202777 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.202989 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.203082 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.204107 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.204109 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.204109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.204118 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.205556 (+ 1438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.205925 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.205927 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.205927 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.205989 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.206198 (+ 209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.206199 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.206199 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.206321 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.206393 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.206547 (+ 154us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.206625 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 15:00:38.206761 15269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.200380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.0.0.1:37230)
0926 15:00:38.200571 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.200581 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.200647 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.200788 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.200788 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.200971 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.201025 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.201027 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.202001 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.202002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.202595 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.202597 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.203083 (+ 486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.203084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.203152 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.203683 (+ 531us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.203689 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.203998 (+ 309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.205659 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.205660 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.205661 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.205840 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.206061 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.206062 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.206063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.206123 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.206383 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.206384 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.206431 (+ 47us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.206515 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.206626 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":100,"threads_started":1}
I20260926 15:00:38.207314 14720 tablet_copy_service.cc:140] P 353bc184be504361bc9d470653da6fdc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 26a22fd6ed974376bb5a6d4da9a14957 ({username='oryx', principal='oryx/127.8.102.61@KRBTEST.COM'} at 127.0.0.1:37230)
I20260926 15:00:38.207444 14720 tablet_copy_service.cc:161] P 353bc184be504361bc9d470653da6fdc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 26a22fd6ed974376bb5a6d4da9a14957 at {username='oryx', principal='oryx/127.8.102.61@KRBTEST.COM'} at 127.0.0.1:37230: session id = 26a22fd6ed974376bb5a6d4da9a14957-00000000000000000000000000000000
I20260926 15:00:38.208269 14720 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 15:00:38.208969 15164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 15:00:38.209960 15164 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: tablet copy: Starting download of 0 data blocks...
I20260926 15:00:38.210134 15164 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: tablet copy: Starting download of 1 WAL segments...
I20260926 15:00:38.211194 15164 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 15:00:38.212165 14720 tablet_copy_service.cc:342] P 353bc184be504361bc9d470653da6fdc: Request end of tablet copy session 26a22fd6ed974376bb5a6d4da9a14957-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.102.61@KRBTEST.COM'} at 127.0.0.1:37230
I20260926 15:00:38.212234 14720 tablet_copy_service.cc:433] P 353bc184be504361bc9d470653da6fdc: ending tablet copy session 26a22fd6ed974376bb5a6d4da9a14957-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 26a22fd6ed974376bb5a6d4da9a14957
I20260926 15:00:38.214377 15164 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:38.215520 15270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:38.215535 15271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:38.215579 15164 server_base.cc:1034] running on GCE node
W20260926 15:00:38.219926 15273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:38.220146 15164 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:38.220379 15164 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:38.221537 15164 hybrid_clock.cc:648] HybridClock initialized: now 1790434838221491 us; error 69 us; skew 500 ppm
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:38.223138 15164 init.cc:377] Logged in from keytab as oryx/127.8.102.61@KRBTEST.COM (short username oryx)
I20260926 15:00:38.223500 15164 webserver.cc:460] Webserver started at http://127.8.102.61:43925/ using document root <none> and password file <none>
I20260926 15:00:38.223635 15164 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:38.223681 15164 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:38.224412 15164 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.001s
I20260926 15:00:38.224957 15280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:38.225128 15164 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 15:00:38.225194 15164 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/wal
uuid: "26a22fd6ed974376bb5a6d4da9a14957"
format_stamp: "Formatted at 2026-09-26 15:00:38 on dist-test-slave-jnhs"
server_key: "1303575623b346032b5941b2780ad938"
server_key_iv: "9613bde1d6c20cf9e342ea442bbec29a"
server_key_version: "encryptionkey@0"
I20260926 15:00:38.225255 15164 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:38.232756 15164 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:38.234622 15164 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:38.237735 15164 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.61:34051
I20260926 15:00:38.238001 15332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.61:34051 every 8 connection(s)
I20260926 15:00:38.238210 15164 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-1/data/info.pb
I20260926 15:00:38.239025 15333 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:00:38.239446 15333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: Bootstrap starting.
I20260926 15:00:38.241616 15333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:00:38.241855 15333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: Bootstrap complete.
I20260926 15:00:38.242084 15333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: NON_VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: true } }
I20260926 15:00:38.242153 15333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 26a22fd6ed974376bb5a6d4da9a14957, State: Initialized, Role: LEARNER
I20260926 15:00:38.242179 15333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: NON_VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: true } }
I20260926 15:00:38.242369 15333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:38.242362 15336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: NON_VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: true } } }
I20260926 15:00:38.242465 15336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [sys.catalog]: This master's current role is: LEARNER
I20260926 15:00:38.244148 15348 catalog_manager.cc:1259] Loaded cluster ID: 509e0d816e344dd3ab0df33248b2f26b
I20260926 15:00:38.244240 15348 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: loading cluster ID for follower catalog manager: success
I20260926 15:00:38.245333 15348 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: acquiring CA information for follower catalog manager: success
I20260926 15:00:38.245682 15348 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 15:00:38.251142 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.243102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49818 (local address 127.8.102.61:34051)
0926 15:00:38.243275 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.243277 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.243286 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:38.243312 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.243313 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.243326 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.243406 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.244144 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.245473 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.246551 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.246874 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.246940 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.248336 (+ 1396us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.248337 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.248337 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.248344 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.249890 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.250349 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.250350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.250350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.250405 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.250618 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.250618 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.250619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.250770 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.250846 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.250996 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.251076 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":330,"server-negotiator.queue_time_us":145,"thread_start_us":108,"threads_started":1}
WARNING: no policy specified for oryx/127.8.102.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.102.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.102.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.60.keytab.
Entry for principal oryx/127.8.102.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.60.keytab.
I20260926 15:00:38.282984 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.60:45261
--webserver_interface=127.8.102.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.60.keytab
--principal=oryx/127.8.102.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:44775,127.8.102.61:34051,127.8.102.60:45261
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:38.378796 15351 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:38.379009 15351 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:38.379032 15351 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:38.380122 15351 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:38.380156 15351 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:38.380170 15351 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:38.380182 15351 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:38.380198 15351 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:00:38.381728 15351 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:44775,127.8.102.61:34051,127.8.102.60:45261
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.60.keytab
--principal=oryx/127.8.102.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.60:45261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.102.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15351
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:38.381917 15351 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:38.382085 15351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 15:00:38.386897 15351 server_base.cc:1034] running on GCE node
W20260926 15:00:38.386864 15357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:38.386859 15359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:38.386874 15356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:38.387323 15351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:38.387717 15351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:38.388886 15351 hybrid_clock.cc:648] HybridClock initialized: now 1790434838388881 us; error 37 us; skew 500 ppm
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:38.391561 15351 init.cc:377] Logged in from keytab as oryx/127.8.102.60@KRBTEST.COM (short username oryx)
I20260926 15:00:38.392022 15351 webserver.cc:460] Webserver started at http://127.8.102.60:43413/ using document root <none> and password file <none>
I20260926 15:00:38.392244 15351 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:38.392307 15351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:38.392452 15351 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:38.393587 15351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/data/instance:
uuid: "98eaf0f289e747c7bc3c658f5fd5a84d"
format_stamp: "Formatted at 2026-09-26 15:00:38 on dist-test-slave-jnhs"
server_key: "9eb43a3e060ccaabb5b585f6e8f2920b"
server_key_iv: "555b8558b6242cb43d4c2c6f42f769b1"
server_key_version: "encryptionkey@0"
I20260926 15:00:38.393945 15351 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/wal/instance:
uuid: "98eaf0f289e747c7bc3c658f5fd5a84d"
format_stamp: "Formatted at 2026-09-26 15:00:38 on dist-test-slave-jnhs"
server_key: "9eb43a3e060ccaabb5b585f6e8f2920b"
server_key_iv: "555b8558b6242cb43d4c2c6f42f769b1"
server_key_version: "encryptionkey@0"
I20260926 15:00:38.395421 15351 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 15:00:38.396247 15366 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:38.396421 15351 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 15:00:38.396512 15351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/wal
uuid: "98eaf0f289e747c7bc3c658f5fd5a84d"
format_stamp: "Formatted at 2026-09-26 15:00:38 on dist-test-slave-jnhs"
server_key: "9eb43a3e060ccaabb5b585f6e8f2920b"
server_key_iv: "555b8558b6242cb43d4c2c6f42f769b1"
server_key_version: "encryptionkey@0"
I20260926 15:00:38.396624 15351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:38.409162 15351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:38.411497 15351 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:38.411664 15351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:38.415333 15351 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.60:45261
I20260926 15:00:38.415472 15418 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.60:45261 every 8 connection(s)
I20260926 15:00:38.415753 15351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/data/info.pb
I20260926 15:00:38.416638 15419 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:38.418188 15419 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:00:38.418805 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 15351
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.60@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434837, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.102.60@KRBTEST.COM
I20260926 15:00:38.428104 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.419188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37244 (local address 127.8.102.62:44775)
0926 15:00:38.419259 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.419261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.419272 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:00:38.419782 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.419784 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.419796 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.419850 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.420403 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.420912 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.421814 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.422023 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.422109 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.424858 (+ 2749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.424861 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.424861 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.424899 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.426547 (+ 1648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.427054 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.427055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.427055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.427114 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.427382 (+ 268us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.427383 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.427384 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.427532 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.427602 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.427918 (+ 316us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.427985 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 15:00:38.428315 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.418907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.0.0.1:37244)
0926 15:00:38.419069 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.419117 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.419677 (+ 560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.419874 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.419877 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.420144 (+ 267us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.420306 (+ 162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.420311 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.421011 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.421012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.421728 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.421730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.422117 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.422118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.422266 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.422952 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.422967 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.424754 (+ 1787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.426622 (+ 1868us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.426627 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.426630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.426971 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.427177 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.427178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.427178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.427312 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.427594 (+ 282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.427598 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.427774 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.427894 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.428075 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":22,"thread_start_us":86,"threads_started":1}
I20260926 15:00:38.428699 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.419375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45180 (local address 127.8.102.60:45261)
0926 15:00:38.419578 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.419583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.419622 (+ 39us) server_negotiation.cc:407] Connection header received
0926 15:00:38.419710 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.419716 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.419734 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.419832 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.420427 (+ 595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.421267 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.421999 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.422360 (+ 361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.422448 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.425243 (+ 2795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.425250 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.425251 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.425258 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.427196 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.427593 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.427596 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.427596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.427651 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.427826 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.427827 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.427827 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.428016 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.428154 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.428478 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.428595 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":172,"thread_start_us":74,"threads_started":1}
I20260926 15:00:38.428996 15419 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.60@KRBTEST.COM for oryx/127.8.102.61@KRBTEST.COM
I20260926 15:00:38.435763 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.429374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49830 (local address 127.8.102.61:34051)
0926 15:00:38.429465 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.429467 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.429475 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:00:38.429506 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.429507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.429515 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.429577 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.429814 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.430299 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.431061 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.431281 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.431360 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.433173 (+ 1813us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.433175 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.433175 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.433183 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.434443 (+ 1260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.434874 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.434875 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.434875 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.434969 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.435216 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.435216 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.435217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.435332 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.435405 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.435577 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.435671 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 15:00:38.435763 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.429289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:34051 (local address 127.0.0.1:49830)
0926 15:00:38.429331 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.429341 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.429375 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.429580 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.429581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.429714 (+ 133us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.429746 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.429747 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.430387 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.430388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.430971 (+ 583us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.430972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.431359 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.431360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.431420 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.431808 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.431813 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.433071 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.434552 (+ 1481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.434553 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.434553 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.434808 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.435076 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.435077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.435077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.435145 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.435395 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.435396 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.435438 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.435552 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.435666 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 15:00:38.436264 15419 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.60@KRBTEST.COM for oryx/127.8.102.60@KRBTEST.COM
I20260926 15:00:38.443691 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.436596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45186 (local address 127.8.102.60:45261)
0926 15:00:38.436642 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.436643 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.436654 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:00:38.436700 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.436700 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.436709 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.436773 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.436971 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.437683 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.438308 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.438620 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.438721 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.440649 (+ 1928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.440651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.440651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.440658 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.442494 (+ 1836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.442888 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.442889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.442889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.442941 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.443165 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.443166 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.443166 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.443282 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.443354 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.443481 (+ 127us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.443576 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":25}
I20260926 15:00:38.443738 15421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.436536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:45261 (local address 127.0.0.1:45186)
0926 15:00:38.436584 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.436611 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.436653 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.436776 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.436776 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.436882 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.436911 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.436911 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.437774 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.437774 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.438229 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.438230 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.438732 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.438733 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.438789 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.439217 (+ 428us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.439221 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.440534 (+ 1313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.442580 (+ 2046us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.442581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.442581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.442806 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.443030 (+ 224us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.443031 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.443031 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.443088 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.443356 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.443357 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.443395 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.443461 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.443622 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":69}
I20260926 15:00:38.445250 15419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d: Bootstrap starting.
I20260926 15:00:38.446143 15419 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:38.446456 15419 log.cc:824] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:38.447293 15419 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d: No bootstrap required, opened a new log
I20260926 15:00:38.448727 15419 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } }
I20260926 15:00:38.448859 15419 raft_consensus.cc:374] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:38.448884 15419 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 98eaf0f289e747c7bc3c658f5fd5a84d, State: Initialized, Role: FOLLOWER
I20260926 15:00:38.449030 15419 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } }
I20260926 15:00:38.449381 15419 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:38.449344 15424 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } } }
I20260926 15:00:38.449543 15424 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [sys.catalog]: This master's current role is: FOLLOWER
W20260926 15:00:38.451143 15435 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:00:38.451220 15435 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:00:38.453680 15351 master_runner.cc:428] Detected that this master 98eaf0f289e747c7bc3c658f5fd5a84d is joining an existing cluster
I20260926 15:00:38.453740 15351 master_runner.cc:432] Initiating AddMaster RPC to add 127.8.102.60:45261
I20260926 15:00:38.462280 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.455613 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49832 (local address 127.8.102.61:34051)
0926 15:00:38.455661 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.455663 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.455880 (+ 217us) server_negotiation.cc:407] Connection header received
0926 15:00:38.455976 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.455977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.455990 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.456023 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.456415 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.457118 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.458161 (+ 1043us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.458457 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.458497 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.459649 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.459651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.459651 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.459658 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.461043 (+ 1385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.461418 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.461420 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.461420 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.461475 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.461670 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.461671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.461672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.461787 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.461856 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.462093 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.462177 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 15:00:38.462298 15441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.455687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:34051 (local address 127.0.0.1:49832)
0926 15:00:38.455822 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.455833 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.455904 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.456096 (+ 192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.456097 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.456267 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.456313 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.456315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.457203 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.457203 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.458061 (+ 858us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.458063 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.458590 (+ 527us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.458591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.458694 (+ 103us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.459191 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.459196 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.459538 (+ 342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.461144 (+ 1606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.461145 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.461146 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.461335 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.461545 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.461546 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.461547 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.461610 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.461848 (+ 238us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.461849 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.461893 (+ 44us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.462059 (+ 166us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.462179 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"mutex_wait_us":3,"thread_start_us":72,"threads_started":1}
I20260926 15:00:38.480250 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 14762
I20260926 15:00:38.486204 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:35343
--local_ip_for_outbound_sockets=127.8.102.1
--tserver_master_addrs=127.8.102.62:44775
--webserver_port=41323
--webserver_interface=127.8.102.1
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.1.keytab
--principal=oryx/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.102.62:44775,127.8.102.61:34051
--tserver_master_addrs=127.8.102.62:44775,127.8.102.61:34051,127.8.102.60:45261 with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:00:38.487123 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.455329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45202 (local address 127.8.102.60:45261)
0926 15:00:38.455370 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.455373 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.455869 (+ 496us) server_negotiation.cc:407] Connection header received
0926 15:00:38.455962 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.455963 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.455971 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.455998 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.456533 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.457248 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.458443 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.458792 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.458841 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.484327 (+ 25486us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.484328 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.484328 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.484334 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.485989 (+ 1655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.486343 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.486344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.486345 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.486388 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.486557 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.486558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.486558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.486710 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.486751 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.486929 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.487021 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 15:00:38.487416 15443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.455269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:45261 (local address 127.0.0.1:45202)
0926 15:00:38.455814 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.455824 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.455898 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.456050 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.456050 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.456389 (+ 339us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.456448 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.456450 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.457334 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.457334 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.458334 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.458337 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.458947 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.458947 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.459032 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.483875 (+ 24843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.483880 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.484230 (+ 350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.486080 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.486080 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.486081 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.486272 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.486449 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.486450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.486450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.486497 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.486790 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.486790 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.486824 (+ 34us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.487233 (+ 409us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.487324 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":71,"threads_started":1}
I20260926 15:00:38.487426 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.455439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37246 (local address 127.8.102.62:44775)
0926 15:00:38.455485 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.455487 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.455501 (+ 14us) server_negotiation.cc:407] Connection header received
0926 15:00:38.455588 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.455589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.455602 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.455673 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.456156 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.456960 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.458161 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.458458 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.458497 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.483842 (+ 25345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.483844 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.483845 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.483855 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.486110 (+ 2255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.486542 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.486543 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.486544 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.486601 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.486824 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.486825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.486825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.486966 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.487006 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.487203 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.487290 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 15:00:38.488050 15442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.455261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.0.0.1:37246)
0926 15:00:38.455449 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.455460 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.455515 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.455730 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.455731 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.455963 (+ 232us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.456058 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.456060 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.457079 (+ 1019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.457080 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.458077 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.458078 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.458590 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.458591 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.458697 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.483230 (+ 24533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.483236 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.483711 (+ 475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.486211 (+ 2500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.486212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.486213 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.486443 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.486697 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.486698 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.486698 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.486758 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.487051 (+ 293us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.487052 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.487094 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.487669 (+ 575us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.487949 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":167,"mutex_wait_us":76,"thread_start_us":142,"threads_started":1}
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434837, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.62@KRBTEST.COM for oryx/127.8.102.60@KRBTEST.COM
I20260926 15:00:38.498500 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.490291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45210 (local address 127.8.102.60:45261)
0926 15:00:38.490330 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.490331 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.490340 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:38.490396 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.490396 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.490406 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.490456 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.490805 (+ 349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.491537 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.492702 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.493023 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.493102 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.495501 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.495504 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.495504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.495511 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.497230 (+ 1719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.497664 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.497665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.497665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.497724 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.497940 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.497941 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.497941 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.498073 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.498141 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.498320 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.498400 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 15:00:38.498574 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.490244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:45261 (local address 127.0.0.1:45210)
0926 15:00:38.490293 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.490304 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.490337 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.490469 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.490470 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.490677 (+ 207us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.490731 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.490733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.491631 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.491632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.492607 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.492610 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.493113 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.493114 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.493195 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.493653 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.493659 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.495416 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.497320 (+ 1904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.497322 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.497323 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.497572 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.497793 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.497794 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.497794 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.497868 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.498147 (+ 279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.498148 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.498202 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.498288 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.498412 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 15:00:38.499480 14700 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.8.102.60:45261: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d"
member_type: NON_VOTER
last_known_addr {
host: "127.8.102.60"
port: 45261
}
attrs {
promote: true
}
}
dest_uuid: "353bc184be504361bc9d470653da6fdc"
cas_config_opid_index: 5
I20260926 15:00:38.499802 14700 consensus_queue.cc:237] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: NON_VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: true } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: NON_VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: true } }
I20260926 15:00:38.500203 15258 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 98eaf0f289e747c7bc3c658f5fd5a84d (127.8.102.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: NON_VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: true } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: NON_VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: true } } }
I20260926 15:00:38.501199 15258 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.8.102.60:45261
I20260926 15:00:38.501276 15258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "353bc184be504361bc9d470653da6fdc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: NON_VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: true } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: NON_VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: true } } }
I20260926 15:00:38.501335 15258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:38.501469 15351 master.cc:537] Master@127.8.102.60:45261 shutting down...
W20260926 15:00:38.501932 14684 consensus_peers.cc:487] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc -> Peer 98eaf0f289e747c7bc3c658f5fd5a84d (127.8.102.60:45261): Couldn't send request to peer 98eaf0f289e747c7bc3c658f5fd5a84d. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260926 15:00:38.505043 15351 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 0 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:38.505153 15351 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 0 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:38.505203 15351 tablet_replica.cc:330] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d: stopping tablet replica
I20260926 15:00:38.506152 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.500308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:34051 (local address 127.0.0.1:49842)
0926 15:00:38.502167 (+ 1859us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.502178 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.502222 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.502424 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.502425 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.502426 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:38.502467 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.502469 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.503456 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.503457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.504721 (+ 1264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.504724 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.505849 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.505850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.505926 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.505952 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.506007 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.506051 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1841}
I20260926 15:00:38.506491 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.500410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49842 (local address 127.8.102.61:34051)
0926 15:00:38.502036 (+ 1626us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.502038 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.502270 (+ 232us) server_negotiation.cc:407] Connection header received
0926 15:00:38.502296 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.502297 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.502320 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.502373 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:38.502554 (+ 181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.503352 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.504829 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.505435 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.505483 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.505560 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.506377 (+ 817us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.506425 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1595}
I20260926 15:00:38.507004 15307 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 1 LEARNER]: Refusing update from remote peer 353bc184be504361bc9d470653da6fdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260926 15:00:38.507565 15259 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: NON_VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260926 15:00:38.508311 15307 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 98eaf0f289e747c7bc3c658f5fd5a84d (127.8.102.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: NON_VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: true } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: NON_VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: true } } }
I20260926 15:00:38.509528 15336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 353bc184be504361bc9d470653da6fdc. Latest consensus state: current_term: 1 leader_uuid: "353bc184be504361bc9d470653da6fdc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: NON_VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: true } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: NON_VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: true } } }
I20260926 15:00:38.509605 15336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [sys.catalog]: This master's current role is: LEARNER
I20260926 15:00:38.509635 15336 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "353bc184be504361bc9d470653da6fdc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: NON_VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: true } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: NON_VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: true } } }
I20260926 15:00:38.509672 15336 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [sys.catalog]: This master's current role is: LEARNER
I20260926 15:00:38.509919 15259 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc: attempting to promote NON_VOTER 26a22fd6ed974376bb5a6d4da9a14957 to VOTER
I20260926 15:00:38.510162 15259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: NON_VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: true } }
I20260926 15:00:38.510710 15307 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 1 LEARNER]: Refusing update from remote peer 353bc184be504361bc9d470653da6fdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260926 15:00:38.510794 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.510569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:45261 (local address 127.0.0.1:45218)
0926 15:00:38.510721 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.510764 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132}
I20260926 15:00:38.510891 15447 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260926 15:00:38.511235 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.511154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:45261 (local address 127.0.0.1:45232)
0926 15:00:38.511187 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.511209 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:45261: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260926 15:00:38.511441 14684 consensus_peers.cc:487] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc -> Peer 98eaf0f289e747c7bc3c658f5fd5a84d (127.8.102.60:45261): Couldn't send request to peer 98eaf0f289e747c7bc3c658f5fd5a84d. Status: Network error: Client connection negotiation failed: client connection to 127.8.102.60:45261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 15:00:38.511521 15258 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 26a22fd6ed974376bb5a6d4da9a14957 (127.8.102.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: NON_VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: true } } }
I20260926 15:00:38.512575 15258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "353bc184be504361bc9d470653da6fdc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: NON_VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: true } } }
I20260926 15:00:38.512668 15258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:38.512727 15258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "353bc184be504361bc9d470653da6fdc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: NON_VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: true } } }
I20260926 15:00:38.512781 15258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:38.513023 15307 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 26a22fd6ed974376bb5a6d4da9a14957 (127.8.102.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: NON_VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: true } } }
I20260926 15:00:38.514010 15446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "353bc184be504361bc9d470653da6fdc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: NON_VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: true } } }
I20260926 15:00:38.514130 15446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:00:38.518889 15351 master.cc:552] Master@127.8.102.60:45261 shutdown complete.
I20260926 15:00:38.518939 15351 master_runner.cc:305] Clearing existing system tablet
I20260926 15:00:38.519083 15351 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:38.519129 15351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:38.519845 15351 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:38.520380 15450 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:38.520563 15351 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:38.520620 15351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:38.520642 15351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/wal
uuid: "98eaf0f289e747c7bc3c658f5fd5a84d"
format_stamp: "Formatted at 2026-09-26 15:00:38 on dist-test-slave-jnhs"
server_key: "9eb43a3e060ccaabb5b585f6e8f2920b"
server_key_iv: "555b8558b6242cb43d4c2c6f42f769b1"
server_key_version: "encryptionkey@0"
I20260926 15:00:38.521318 15351 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:38.523434 15351 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260926 15:00:38.523583 15351 log.cc:1196] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260926 15:00:38.523837 15351 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d: Deleting consensus metadata
I20260926 15:00:38.523985 15351 master_runner.cc:315] Copying system tablet from 127.8.102.62:44775
I20260926 15:00:38.524622 15351 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d: tablet copy: Beginning tablet copy session from remote peer at address 127.8.102.62:44775
I20260926 15:00:38.532413 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.525084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37248 (local address 127.8.102.62:44775)
0926 15:00:38.525132 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.525134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.525280 (+ 146us) server_negotiation.cc:407] Connection header received
0926 15:00:38.525387 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.525388 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.525401 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.525483 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.525888 (+ 405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.526705 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.527811 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.528103 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.528177 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.529377 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.529378 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.529379 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.529388 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.531073 (+ 1685us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.531506 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.531507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.531508 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.531576 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.531842 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.531842 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.531843 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.531977 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.532045 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.532239 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.532316 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 15:00:38.532435 15456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.525003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.0.0.1:37248)
0926 15:00:38.525193 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.525206 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.525276 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.525499 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.525500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.525722 (+ 222us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.525796 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.525798 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.526809 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.526811 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.527712 (+ 901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.527715 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.528197 (+ 482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.528198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.528277 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.528796 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.528802 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.529276 (+ 474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.531159 (+ 1883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.531161 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.531161 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.531411 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.531649 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.531650 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.531651 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.531728 (+ 77us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.532056 (+ 328us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.532058 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.532113 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.532204 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.532328 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":169,"thread_start_us":69,"threads_started":1}
I20260926 15:00:38.532977 14720 tablet_copy_service.cc:140] P 353bc184be504361bc9d470653da6fdc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 98eaf0f289e747c7bc3c658f5fd5a84d ({username='oryx', principal='oryx/127.8.102.60@KRBTEST.COM'} at 127.0.0.1:37248)
I20260926 15:00:38.533032 14720 tablet_copy_service.cc:161] P 353bc184be504361bc9d470653da6fdc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 98eaf0f289e747c7bc3c658f5fd5a84d at {username='oryx', principal='oryx/127.8.102.60@KRBTEST.COM'} at 127.0.0.1:37248: session id = 98eaf0f289e747c7bc3c658f5fd5a84d-00000000000000000000000000000000
I20260926 15:00:38.533607 14720 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc: Tablet Copy: opened 0 blocks and 1 log segments
I20260926 15:00:38.534098 15351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260926 15:00:38.535496 15351 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d: tablet copy: Starting download of 0 data blocks...
I20260926 15:00:38.535730 15351 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d: tablet copy: Starting download of 1 WAL segments...
I20260926 15:00:38.536537 15351 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260926 15:00:38.537597 14720 tablet_copy_service.cc:342] P 353bc184be504361bc9d470653da6fdc: Request end of tablet copy session 98eaf0f289e747c7bc3c658f5fd5a84d-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.8.102.60@KRBTEST.COM'} at 127.0.0.1:37248
I20260926 15:00:38.537659 14720 tablet_copy_service.cc:433] P 353bc184be504361bc9d470653da6fdc: ending tablet copy session 98eaf0f289e747c7bc3c658f5fd5a84d-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 98eaf0f289e747c7bc3c658f5fd5a84d
I20260926 15:00:38.540584 15351 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:38.541833 15457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:38.543402 15458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:38.543689 15351 server_base.cc:1034] running on GCE node
W20260926 15:00:38.543862 15460 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:38.544073 15351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:38.544204 15351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:38.545318 15351 hybrid_clock.cc:648] HybridClock initialized: now 1790434838545310 us; error 28 us; skew 500 ppm
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:38.547129 15351 init.cc:377] Logged in from keytab as oryx/127.8.102.60@KRBTEST.COM (short username oryx)
I20260926 15:00:38.547397 15351 webserver.cc:460] Webserver started at http://127.8.102.60:37593/ using document root <none> and password file <none>
I20260926 15:00:38.547509 15351 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:38.547546 15351 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:38.548264 15351 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260926 15:00:38.548844 15467 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:38.548969 15351 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 15:00:38.549011 15351 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/wal
uuid: "98eaf0f289e747c7bc3c658f5fd5a84d"
format_stamp: "Formatted at 2026-09-26 15:00:38 on dist-test-slave-jnhs"
server_key: "9eb43a3e060ccaabb5b585f6e8f2920b"
server_key_iv: "555b8558b6242cb43d4c2c6f42f769b1"
server_key_version: "encryptionkey@0"
I20260926 15:00:38.549077 15351 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:38.562321 15351 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:38.564245 15351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:38.567350 15351 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.60:45261
I20260926 15:00:38.567742 15351 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/master-2/data/info.pb
I20260926 15:00:38.567642 15519 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.60:45261 every 8 connection(s)
I20260926 15:00:38.568723 15520 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:00:38.569298 15520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d: Bootstrap starting.
I20260926 15:00:38.572657 15520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:00:38.573028 15520 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d: Bootstrap complete.
I20260926 15:00:38.573321 15520 raft_consensus.cc:348] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: NON_VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: true } }
I20260926 15:00:38.573534 15520 raft_consensus.cc:729] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 98eaf0f289e747c7bc3c658f5fd5a84d, State: Initialized, Role: LEARNER
I20260926 15:00:38.573628 15520 consensus_queue.cc:260] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: NON_VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: true } }
I20260926 15:00:38.573954 15520 sys_catalog.cc:564] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:38.573966 15523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: NON_VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: true } } }
I20260926 15:00:38.574106 15523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [sys.catalog]: This master's current role is: LEARNER
I20260926 15:00:38.575806 15534 catalog_manager.cc:1259] Loaded cluster ID: 509e0d816e344dd3ab0df33248b2f26b
I20260926 15:00:38.575891 15534 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d: loading cluster ID for follower catalog manager: success
I20260926 15:00:38.577406 15534 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d: acquiring CA information for follower catalog manager: success
I20260926 15:00:38.577713 15534 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260926 15:00:38.593500 15444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:38.593724 15444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:38.593750 15444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:38.595000 15444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:38.595050 15444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:38.595079 15444 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:00:38.596577 15444 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.1.keytab
--principal=oryx/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:35343
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=41323
--tserver_master_addrs=127.8.102.62:44775,127.8.102.61:34051,127.8.102.60:45261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15444
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:38.596815 15444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:38.597064 15444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:38.601943 15539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:38.601943 15540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:38.601969 15542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:38.602317 15444 server_base.cc:1034] running on GCE node
I20260926 15:00:38.602509 15444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:38.602875 15444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:38.604038 15444 hybrid_clock.cc:648] HybridClock initialized: now 1790434838604018 us; error 35 us; skew 500 ppm
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:38.606581 15444 init.cc:377] Logged in from keytab as oryx/127.8.102.1@KRBTEST.COM (short username oryx)
I20260926 15:00:38.607021 15444 webserver.cc:460] Webserver started at http://127.8.102.1:41323/ using document root <none> and password file <none>
I20260926 15:00:38.607257 15444 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:38.607314 15444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:38.608677 15444 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:38.609309 15549 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:38.609503 15444 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:38.609592 15444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "9425f4a721a6474bb40adb111db99615"
format_stamp: "Formatted at 2026-09-26 15:00:37 on dist-test-slave-jnhs"
server_key: "944691509257e51fdd43de592e951a54"
server_key_iv: "8e0a1db909e68d8b0d34c07e0c21353f"
server_key_version: "encryptionkey@0"
I20260926 15:00:38.609947 15444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:38.627888 15444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:38.630194 15444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:38.630378 15444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:38.630683 15444 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:38.631043 15444 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:38.631102 15444 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:38.631145 15444 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:38.631174 15444 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:38.637869 15444 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:35343
I20260926 15:00:38.637902 15662 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:35343 every 8 connection(s)
I20260926 15:00:38.638324 15444 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:00:38.643518 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 15444
I20260926 15:00:38.643699 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 14896
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.60@KRBTEST.COM
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.61@KRBTEST.COM
I20260926 15:00:38.651014 15668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.639186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:45261 (local address 127.8.102.1:35129)
0926 15:00:38.639396 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.639435 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.640123 (+ 688us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.640413 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.640417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.640773 (+ 356us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.641014 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.641019 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.642341 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.642341 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.643228 (+ 887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.643230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.643798 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.643800 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.643966 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.644600 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.644609 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.646524 (+ 1915us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.648904 (+ 2380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.648912 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.648915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.649278 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.649556 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.649557 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.649557 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.649779 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.650126 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.650133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.650331 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.650457 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.650692 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":158,"mutex_wait_us":2,"thread_start_us":97,"threads_started":1}
I20260926 15:00:38.651511 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:33887
--local_ip_for_outbound_sockets=127.8.102.2
--tserver_master_addrs=127.8.102.62:44775
--webserver_port=36803
--webserver_interface=127.8.102.2
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.2.keytab
--principal=oryx/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.102.62:44775,127.8.102.61:34051
--tserver_master_addrs=127.8.102.62:44775,127.8.102.61:34051,127.8.102.60:45261 with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:00:38.651865 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.639673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:57545 (local address 127.8.102.62:44775)
0926 15:00:38.639724 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.639726 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.639785 (+ 59us) server_negotiation.cc:407] Connection header received
0926 15:00:38.640270 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.640271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.640285 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.640322 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.641139 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.641803 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.643339 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.643706 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.643786 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.647513 (+ 3727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.647515 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.647516 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.647524 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.649848 (+ 2324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.650297 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.650298 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.650299 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.650360 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.650922 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.650923 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.650923 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.651350 (+ 427us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.651421 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.651658 (+ 237us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.651741 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 15:00:38.652275 15670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.639588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.1:57545)
0926 15:00:38.639717 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.639728 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.640149 (+ 421us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.640476 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.640478 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.640822 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.641017 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.641019 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.641927 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.641928 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.643211 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.643214 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.644553 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.644554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.644638 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.645161 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.645167 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.647410 (+ 2243us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.649951 (+ 2541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.649952 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.649952 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.650211 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.650763 (+ 552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.650764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.650764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.650836 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.651481 (+ 645us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.651481 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.651534 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.652010 (+ 476us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.652134 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":22,"thread_start_us":65,"threads_started":1}
I20260926 15:00:38.652510 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.639187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:35129 (local address 127.8.102.60:45261)
0926 15:00:38.639312 (+ 125us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.639314 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.639530 (+ 216us) server_negotiation.cc:407] Connection header received
0926 15:00:38.640245 (+ 715us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.640246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.640261 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.640327 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.641115 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.642257 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.643321 (+ 1064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.643665 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.643735 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.646669 (+ 2934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.646671 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.646672 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.646681 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.648790 (+ 2109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.649375 (+ 585us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.649376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.649376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.649484 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.649868 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.649869 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.649870 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.650033 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.650097 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.651107 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.651239 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
I20260926 15:00:38.653118 15664 heartbeater.cc:347] Connected to a master server at 127.8.102.62:44775
I20260926 15:00:38.653213 15663 heartbeater.cc:347] Connected to a master server at 127.8.102.60:45261
I20260926 15:00:38.653231 15664 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:38.653249 15663 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:38.653450 15664 heartbeater.cc:511] Master 127.8.102.62:44775 requested a full tablet report, sending...
I20260926 15:00:38.653447 15663 heartbeater.cc:511] Master 127.8.102.60:45261 requested a full tablet report, sending...
I20260926 15:00:38.653761 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.639253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:48769 (local address 127.8.102.61:34051)
0926 15:00:38.639305 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.639308 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.639770 (+ 462us) server_negotiation.cc:407] Connection header received
0926 15:00:38.640269 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.640271 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.640285 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.640322 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.641212 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.642068 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.647388 (+ 5320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.647746 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.647795 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.650180 (+ 2385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.650182 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.650182 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.650191 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.652019 (+ 1828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.652746 (+ 727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.652748 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.652748 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.652834 (+ 86us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.653080 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.653081 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.653081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.653247 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.653294 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.653534 (+ 240us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.653632 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 15:00:38.653952 14700 ts_manager.cc:194] Re-registered known tserver with Master: 9425f4a721a6474bb40adb111db99615 (127.8.102.1:35343)
I20260926 15:00:38.654057 15484 ts_manager.cc:194] Registered new tserver with Master: 9425f4a721a6474bb40adb111db99615 (127.8.102.1:35343)
I20260926 15:00:38.654157 15669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.639186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:34051 (local address 127.8.102.1:48769)
0926 15:00:38.639695 (+ 509us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.639708 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.640149 (+ 441us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.640834 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.640835 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.641024 (+ 189us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.641084 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.641086 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.642164 (+ 1078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.642165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.643192 (+ 1027us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.643194 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.647899 (+ 4705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.647900 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.647988 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.648393 (+ 405us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.648399 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.650073 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.652111 (+ 2038us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.652112 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.652113 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.652337 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.652921 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.652923 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.652923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.653001 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.653345 (+ 344us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.653346 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.653407 (+ 61us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.653883 (+ 476us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.654014 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"mutex_wait_us":32,"thread_start_us":84,"threads_started":1}
I20260926 15:00:38.654505 14700 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:57545
I20260926 15:00:38.654986 15665 heartbeater.cc:347] Connected to a master server at 127.8.102.61:34051
I20260926 15:00:38.655032 15665 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:38.655100 15665 heartbeater.cc:511] Master 127.8.102.61:34051 requested a full tablet report, sending...
I20260926 15:00:38.655504 15297 ts_manager.cc:194] Registered new tserver with Master: 9425f4a721a6474bb40adb111db99615 (127.8.102.1:35343)
W20260926 15:00:38.742969 15671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:38.743227 15671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:38.743283 15671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:38.744601 15671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:38.744647 15671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:38.744673 15671 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:00:38.746176 15671 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.2.keytab
--principal=oryx/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:33887
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=36803
--tserver_master_addrs=127.8.102.62:44775,127.8.102.61:34051,127.8.102.60:45261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15671
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:38.746404 15671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:38.746591 15671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:38.751895 15676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:38.751914 15679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:38.751916 15671 server_base.cc:1034] running on GCE node
W20260926 15:00:38.752071 15677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:38.752408 15671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:38.752696 15671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:38.753847 15671 hybrid_clock.cc:648] HybridClock initialized: now 1790434838753835 us; error 26 us; skew 500 ppm
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:38.756454 15671 init.cc:377] Logged in from keytab as oryx/127.8.102.2@KRBTEST.COM (short username oryx)
I20260926 15:00:38.756892 15671 webserver.cc:460] Webserver started at http://127.8.102.2:36803/ using document root <none> and password file <none>
I20260926 15:00:38.757089 15671 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:38.757174 15671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:38.758566 15671 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:38.759199 15686 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:38.759383 15671 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 15:00:38.759482 15671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "5d53cf9c2da74a33bb309bdea1008cd5"
format_stamp: "Formatted at 2026-09-26 15:00:37 on dist-test-slave-jnhs"
server_key: "784191f64e1ee5873a10b8ff521fe825"
server_key_iv: "7e444ae663f9632d7eaaad0a4a00a133"
server_key_version: "encryptionkey@0"
I20260926 15:00:38.759835 15671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:38.771972 15671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:38.774330 15671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:38.774493 15671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:38.774791 15671 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:38.775117 15671 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:38.775172 15671 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:38.775204 15671 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:38.775221 15671 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:38.781268 15671 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:33887
I20260926 15:00:38.781332 15799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:33887 every 8 connection(s)
I20260926 15:00:38.781670 15671 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:00:38.787657 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 15671
I20260926 15:00:38.787842 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 15030
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.60@KRBTEST.COM
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.61@KRBTEST.COM
I20260926 15:00:38.795252 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.782550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:46263 (local address 127.8.102.62:44775)
0926 15:00:38.782612 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.782614 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.783152 (+ 538us) server_negotiation.cc:407] Connection header received
0926 15:00:38.784273 (+ 1121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.784275 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.784288 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.784343 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.785093 (+ 750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.785635 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.787126 (+ 1491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.787325 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.787358 (+ 33us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.791710 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.791712 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.791712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.791721 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.793512 (+ 1791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.794051 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.794053 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.794053 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.794113 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.794400 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.794401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.794402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.794568 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.794621 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.795012 (+ 391us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.795104 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 15:00:38.795251 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.782856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:44019 (local address 127.8.102.61:34051)
0926 15:00:38.782933 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.782935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.783166 (+ 231us) server_negotiation.cc:407] Connection header received
0926 15:00:38.784295 (+ 1129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.784297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.784310 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.784348 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.785108 (+ 760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.785966 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.787114 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.787444 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.787518 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.791727 (+ 4209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.791729 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.791729 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.791738 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.793857 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.794289 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.794290 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.794290 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.794351 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.794595 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.794596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.794596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.794766 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.794815 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.795044 (+ 229us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.795125 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 15:00:38.795622 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.782610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:58327 (local address 127.8.102.60:45261)
0926 15:00:38.782658 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.782660 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.783163 (+ 503us) server_negotiation.cc:407] Connection header received
0926 15:00:38.784295 (+ 1132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.784297 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.784314 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.784350 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.785118 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.785957 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.787609 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.787945 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.787993 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.791210 (+ 3217us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.791212 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.791213 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.791222 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.794265 (+ 3043us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.794709 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.794710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.794710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.794765 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.795000 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.795001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.795002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.795140 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.795196 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.795416 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.795513 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 15:00:38.796025 15805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.782434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.2:46263)
0926 15:00:38.783019 (+ 585us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.783056 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.784163 (+ 1107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.784419 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.784424 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.784786 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.784991 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.784996 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.785735 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.785736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.787006 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.787008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.787453 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.787454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.787616 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.788399 (+ 783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.788410 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.791599 (+ 3189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.793620 (+ 2021us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.793627 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.793630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.793966 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.794186 (+ 220us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.794187 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.794187 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.794326 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.794676 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.794682 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.794857 (+ 175us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.795374 (+ 517us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.795704 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"mutex_wait_us":154,"thread_start_us":86,"threads_started":1}
I20260926 15:00:38.796025 15804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.782434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:45261 (local address 127.8.102.2:58327)
0926 15:00:38.783005 (+ 571us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.783061 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.784155 (+ 1094us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.784446 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.784447 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.784835 (+ 388us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.784992 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.785000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.786720 (+ 1720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.786721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.787508 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.787511 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.788078 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.788079 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.788146 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.788786 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.788791 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.791075 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.794352 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.794353 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.794354 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.794591 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.794842 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.794843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.794843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.794901 (+ 58us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.795231 (+ 330us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.795232 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.795285 (+ 53us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.795706 (+ 421us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.795810 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"mutex_wait_us":29,"thread_start_us":119,"threads_started":1}
I20260926 15:00:38.796025 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.782796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:34051 (local address 127.8.102.2:44019)
0926 15:00:38.783103 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.783113 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.784155 (+ 1042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.784423 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.784424 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.784807 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.784992 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.785000 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.786062 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.786062 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.787006 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.787008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.787575 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.787576 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.787656 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.788463 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.788470 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.791593 (+ 3123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.793962 (+ 2369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.793964 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.793964 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.794194 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.794436 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.794436 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.794437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.794510 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.794868 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.794869 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.794914 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.795381 (+ 467us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.795615 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":285,"mutex_wait_us":11,"thread_start_us":243,"threads_started":1}
I20260926 15:00:38.796818 15801 heartbeater.cc:347] Connected to a master server at 127.8.102.62:44775
I20260926 15:00:38.796834 15802 heartbeater.cc:347] Connected to a master server at 127.8.102.61:34051
I20260926 15:00:38.796932 15801 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:38.796934 15802 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:38.797127 15802 heartbeater.cc:511] Master 127.8.102.61:34051 requested a full tablet report, sending...
I20260926 15:00:38.797132 15801 heartbeater.cc:511] Master 127.8.102.62:44775 requested a full tablet report, sending...
I20260926 15:00:38.797312 15800 heartbeater.cc:347] Connected to a master server at 127.8.102.60:45261
I20260926 15:00:38.797355 15800 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:38.797425 15800 heartbeater.cc:511] Master 127.8.102.60:45261 requested a full tablet report, sending...
I20260926 15:00:38.797540 14700 ts_manager.cc:194] Re-registered known tserver with Master: 5d53cf9c2da74a33bb309bdea1008cd5 (127.8.102.2:33887)
I20260926 15:00:38.797569 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:36979
--local_ip_for_outbound_sockets=127.8.102.3
--tserver_master_addrs=127.8.102.62:44775
--webserver_port=32847
--webserver_interface=127.8.102.3
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.3.keytab
--principal=oryx/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.8.102.62:44775,127.8.102.61:34051
--tserver_master_addrs=127.8.102.62:44775,127.8.102.61:34051,127.8.102.60:45261 with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:00:38.797663 15484 ts_manager.cc:194] Registered new tserver with Master: 5d53cf9c2da74a33bb309bdea1008cd5 (127.8.102.2:33887)
I20260926 15:00:38.797663 15297 ts_manager.cc:194] Registered new tserver with Master: 5d53cf9c2da74a33bb309bdea1008cd5 (127.8.102.2:33887)
I20260926 15:00:38.798087 14700 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:46263
W20260926 15:00:38.890087 15807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:38.890327 15807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:38.890357 15807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:38.891654 15807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:38.891727 15807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:38.891788 15807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:00:38.893837 15807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.3.keytab
--principal=oryx/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:36979
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=32847
--tserver_master_addrs=127.8.102.62:44775,127.8.102.61:34051,127.8.102.60:45261
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.15807
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:38.894057 15807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:38.894271 15807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:38.899844 15813 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:38.899857 15815 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:38.899860 15812 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:38.900036 15807 server_base.cc:1034] running on GCE node
I20260926 15:00:38.900367 15807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:38.900609 15807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:38.901763 15807 hybrid_clock.cc:648] HybridClock initialized: now 1790434838901747 us; error 33 us; skew 500 ppm
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:38.904736 15807 init.cc:377] Logged in from keytab as oryx/127.8.102.3@KRBTEST.COM (short username oryx)
I20260926 15:00:38.905202 15807 webserver.cc:460] Webserver started at http://127.8.102.3:32847/ using document root <none> and password file <none>
I20260926 15:00:38.905402 15807 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:38.905468 15807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:38.906975 15807 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:38.907596 15822 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:38.907779 15807 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:38.907889 15807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "e82ec9077f6244fa83a17bfcfe8af69a"
format_stamp: "Formatted at 2026-09-26 15:00:37 on dist-test-slave-jnhs"
server_key: "b297f0fdfe24eb5109161d9149d325bc"
server_key_iv: "60c22a5a8fd9e04b615631e89a2bb433"
server_key_version: "encryptionkey@0"
I20260926 15:00:38.908288 15807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:38.939931 15807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:38.942078 15807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:38.942250 15807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:38.942548 15807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:38.942929 15807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:38.942991 15807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:38.943041 15807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:38.943105 15807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:38.949388 15807 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:36979
I20260926 15:00:38.949491 15935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:36979 every 8 connection(s)
I20260926 15:00:38.949824 15807 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:00:38.953716 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 15807
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.61@KRBTEST.COM
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.60@KRBTEST.COM
Sep 26 15:00:38 dist-test-slave-jnhs krb5kdc[14655](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434838, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM
I20260926 15:00:38.962090 15341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.951187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:57837 (local address 127.8.102.61:34051)
0926 15:00:38.951232 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.951234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.951242 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:00:38.951603 (+ 361us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.951604 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.951617 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.951671 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.952363 (+ 692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.952864 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.954194 (+ 1330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.954499 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.954559 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.958781 (+ 4222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.958783 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.958783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.958790 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.960420 (+ 1630us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.960921 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.960922 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.960923 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.960987 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.961277 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.961279 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.961279 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.961421 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.961483 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.961872 (+ 389us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.961950 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 15:00:38.962826 15941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.950800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:34051 (local address 127.8.102.3:57837)
0926 15:00:38.950920 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.950931 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.951484 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.951746 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.951750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.952055 (+ 305us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.952264 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.952268 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.952973 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.952974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.954092 (+ 1118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.954095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.954698 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.954699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.954873 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.955416 (+ 543us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.955428 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.958683 (+ 3255us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.960507 (+ 1824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.960523 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.960525 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.960838 (+ 313us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.961066 (+ 228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.961067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.961067 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.961198 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.961513 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.961520 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.961705 (+ 185us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.962203 (+ 498us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.962449 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":78,"threads_started":1}
I20260926 15:00:38.963922 15938 heartbeater.cc:347] Connected to a master server at 127.8.102.61:34051
I20260926 15:00:38.964031 15938 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:38.964243 15938 heartbeater.cc:511] Master 127.8.102.61:34051 requested a full tablet report, sending...
I20260926 15:00:38.964797 15297 ts_manager.cc:194] Registered new tserver with Master: e82ec9077f6244fa83a17bfcfe8af69a (127.8.102.3:36979)
I20260926 15:00:38.964999 15667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.950548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:42235 (local address 127.8.102.60:45261)
0926 15:00:38.950620 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.950622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.950938 (+ 316us) server_negotiation.cc:407] Connection header received
0926 15:00:38.951637 (+ 699us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.951639 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.951653 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.951686 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.952379 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.953141 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.954341 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.954663 (+ 322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.954707 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.961832 (+ 7125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.961834 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.961834 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.961842 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.963615 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.964151 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.964152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.964153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.964209 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.964410 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.964411 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.964412 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.964549 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.964593 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.964793 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.964891 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 15:00:38.965333 15940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.950578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:45261 (local address 127.8.102.3:42235)
0926 15:00:38.950783 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.950822 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.951520 (+ 698us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.951759 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.951760 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.952123 (+ 363us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.952264 (+ 141us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.952271 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.953242 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.953243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.954233 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.954236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.954793 (+ 557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.954793 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.955049 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.955980 (+ 931us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.955986 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.961683 (+ 5697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.963767 (+ 2084us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.963768 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.963769 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.964061 (+ 292us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.964282 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.964283 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.964283 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.964340 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.964642 (+ 302us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.964642 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.964685 (+ 43us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.965115 (+ 430us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.965232 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"mutex_wait_us":78,"thread_start_us":106,"threads_started":1}
I20260926 15:00:38.965651 15936 heartbeater.cc:347] Connected to a master server at 127.8.102.60:45261
I20260926 15:00:38.965706 15936 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:38.965782 15936 heartbeater.cc:511] Master 127.8.102.60:45261 requested a full tablet report, sending...
I20260926 15:00:38.966100 15484 ts_manager.cc:194] Registered new tserver with Master: e82ec9077f6244fa83a17bfcfe8af69a (127.8.102.3:36979)
I20260926 15:00:38.967123 14745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.951047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:47657 (local address 127.8.102.62:44775)
0926 15:00:38.951089 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.951091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.951207 (+ 116us) server_negotiation.cc:407] Connection header received
0926 15:00:38.951638 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.951640 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.951653 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.951686 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.952392 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.953140 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.954342 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.954719 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.954761 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.961802 (+ 7041us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.961804 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.961805 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.961814 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.964374 (+ 2560us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.966097 (+ 1723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.966099 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.966099 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.966153 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.966349 (+ 196us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.966350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.966351 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.966497 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.966565 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.966948 (+ 383us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.967027 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 15:00:38.967499 15942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.950993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.3:47657)
0926 15:00:38.951141 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.951151 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.951515 (+ 364us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.951761 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.951761 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.952215 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:38.952266 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.952271 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.953242 (+ 971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.953243 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.954233 (+ 990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.954236 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.954842 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.954843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.955352 (+ 509us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.956678 (+ 1326us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:38.956683 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:38.961462 (+ 4779us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:38.965296 (+ 3834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.965297 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.965297 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.965535 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.966226 (+ 691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:38.966227 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:38.966228 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:38.966285 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:38.966777 (+ 492us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:38.966778 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:38.966823 (+ 45us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.967256 (+ 433us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.967368 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"mutex_wait_us":476,"thread_start_us":70,"threads_started":1}
I20260926 15:00:38.967706 15947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.954902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37260 (local address 127.8.102.62:44775)
0926 15:00:38.955054 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.955056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.956594 (+ 1538us) server_negotiation.cc:407] Connection header received
0926 15:00:38.956620 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.956621 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.956636 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.956703 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.957891 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.958705 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.959677 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.960749 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.960816 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.964213 (+ 3397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.964215 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.964215 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.964223 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.965959 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.966613 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.966615 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.966615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.966695 (+ 80us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.966895 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.966896 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.966896 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.967027 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.967119 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.967507 (+ 388us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.967588 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":102,"threads_started":1}
I20260926 15:00:38.968187 15937 heartbeater.cc:347] Connected to a master server at 127.8.102.62:44775
I20260926 15:00:38.968242 15937 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:38.968312 15937 heartbeater.cc:511] Master 127.8.102.62:44775 requested a full tablet report, sending...
I20260926 15:00:38.968688 14700 ts_manager.cc:194] Re-registered known tserver with Master: e82ec9077f6244fa83a17bfcfe8af69a (127.8.102.3:36979)
I20260926 15:00:38.968924 15948 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.955084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45238 (local address 127.8.102.60:45261)
0926 15:00:38.955210 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.955212 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.955409 (+ 197us) server_negotiation.cc:407] Connection header received
0926 15:00:38.955526 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.955527 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.955541 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.955580 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.956390 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.957163 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.958077 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.958916 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.958975 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.960808 (+ 1833us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.960810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.960810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.960818 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.965208 (+ 4390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.965947 (+ 739us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.965948 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.965948 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.966005 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.966551 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.966552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.966553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.966710 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.968695 (+ 1985us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.968742 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.968825 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":81,"threads_started":1}
I20260926 15:00:38.969141 14700 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:47657
I20260926 15:00:38.969445 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.958399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:45261 (local address 127.0.0.1:45252)
0926 15:00:38.958473 (+ 74us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.958482 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.958515 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.962796 (+ 4281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.962797 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.962800 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:38.962850 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.962853 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.967233 (+ 4380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.967234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.968399 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.968402 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.969249 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.969251 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.969309 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.969313 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.969344 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.969382 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20260926 15:00:38.969462 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.955689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49846 (local address 127.8.102.61:34051)
0926 15:00:38.956418 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.956420 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.956815 (+ 395us) server_negotiation.cc:407] Connection header received
0926 15:00:38.957137 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.957138 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.957152 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.957191 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:38.958565 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.959310 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.960395 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.960729 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.960789 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.963465 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:38.963466 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:38.963466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:38.963473 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:38.967484 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.968146 (+ 662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.968148 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.968148 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.968205 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:38.968547 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:38.968548 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:38.968548 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:38.968683 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:38.968805 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.969263 (+ 458us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.969369 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":710,"thread_start_us":50,"threads_started":1}
I20260926 15:00:38.969846 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.958582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45252 (local address 127.8.102.60:45261)
0926 15:00:38.962129 (+ 3547us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.962131 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.962138 (+ 7us) server_negotiation.cc:407] Connection header received
0926 15:00:38.962237 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.962238 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.962290 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.966297 (+ 4007us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:38.966334 (+ 37us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.967117 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.968493 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.969153 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.969618 (+ 465us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.969714 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.969737 (+ 23us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.969784 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3518,"thread_start_us":73,"threads_started":1}
I20260926 15:00:38.970113 15494 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260926 15:00:38.970316 15523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 353bc184be504361bc9d470653da6fdc. Latest consensus state: current_term: 1 leader_uuid: "353bc184be504361bc9d470653da6fdc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: NON_VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: true } } }
I20260926 15:00:38.970410 15523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [sys.catalog]: This master's current role is: LEARNER
I20260926 15:00:38.970844 14700 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37260:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:00:38.972409 14700 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:00:38.973104 15259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "353bc184be504361bc9d470653da6fdc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: NON_VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: true } } }
I20260926 15:00:38.973219 15259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:38.974653 15259 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc: attempting to promote NON_VOTER 98eaf0f289e747c7bc3c658f5fd5a84d to VOTER
I20260926 15:00:38.974776 15259 consensus_queue.cc:237] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false } }
I20260926 15:00:38.975214 15494 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 1 LEARNER]: Refusing update from remote peer 353bc184be504361bc9d470653da6fdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 15:00:38.975490 15258 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 15:00:38.975880 15307 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 1 FOLLOWER]: Refusing update from remote peer 353bc184be504361bc9d470653da6fdc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260926 15:00:38.976090 15448 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [LEADER]: Connected to new peer: Peer: permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260926 15:00:38.976253 15259 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 98eaf0f289e747c7bc3c658f5fd5a84d (127.8.102.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false } } }
I20260926 15:00:38.976573 15494 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 98eaf0f289e747c7bc3c658f5fd5a84d (127.8.102.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false } } }
I20260926 15:00:38.976953 15307 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 98eaf0f289e747c7bc3c658f5fd5a84d (127.8.102.60) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false } } }
I20260926 15:00:38.977330 15259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "353bc184be504361bc9d470653da6fdc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false } } }
I20260926 15:00:38.977401 15259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:38.977370 15258 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "353bc184be504361bc9d470653da6fdc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false } } }
I20260926 15:00:38.977471 15259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "353bc184be504361bc9d470653da6fdc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false } } }
I20260926 15:00:38.977523 15259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:38.977483 15258 sys_catalog.cc:458] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:38.977685 15523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "353bc184be504361bc9d470653da6fdc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false } } }
I20260926 15:00:38.977779 15523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:00:38.978000 15446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "353bc184be504361bc9d470653da6fdc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false } } }
I20260926 15:00:38.978085 15446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:00:38.982933 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.978783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38776 (local address 127.8.102.3:36979)
0926 15:00:38.979284 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.979288 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.979302 (+ 14us) server_negotiation.cc:407] Connection header received
0926 15:00:38.979379 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.979381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.979396 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.979473 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:38.979609 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.980920 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.981855 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.982541 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.982625 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.982722 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.982819 (+ 97us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.982854 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":464,"thread_start_us":79,"threads_started":1}
I20260926 15:00:38.982995 15257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.978572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:33887 (local address 127.0.0.1:38052)
0926 15:00:38.978676 (+ 104us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.978688 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.978733 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.979359 (+ 626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.979360 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.979362 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:38.979404 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.979406 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.980670 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.980672 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.981773 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.981775 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.982740 (+ 965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.982741 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.982803 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.982808 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.982841 (+ 33us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.982887 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":4}
I20260926 15:00:38.982941 15965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.979110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:36979 (local address 127.0.0.1:38776)
0926 15:00:38.979255 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.979264 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.979310 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.979476 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.979477 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.979479 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:38.979511 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.979513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.980984 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.980985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.981749 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.981751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.982660 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.982661 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.982740 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.982745 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.982802 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.982847 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":2,"thread_start_us":81,"threads_started":1}
I20260926 15:00:38.983151 15961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.978687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:35343 (local address 127.0.0.1:44932)
0926 15:00:38.978929 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:38.978939 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:38.978984 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:38.979322 (+ 338us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:38.979323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:38.979326 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:38.979383 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.979385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.980947 (+ 1562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.980948 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.982024 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:38.982026 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.982932 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.982933 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:38.982991 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.982995 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:00:38.983025 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 15:00:38.983074 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"mutex_wait_us":2,"thread_start_us":96,"threads_started":1}
I20260926 15:00:38.983747 15962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.978728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38052 (local address 127.8.102.2:33887)
0926 15:00:38.979165 (+ 437us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.979169 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.979176 (+ 7us) server_negotiation.cc:407] Connection header received
0926 15:00:38.979232 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.979234 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.979251 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.979494 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:38.979557 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.980585 (+ 1028us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.981869 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.982610 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.982981 (+ 371us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.983585 (+ 604us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.983610 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.983664 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":102,"threads_started":1}
I20260926 15:00:38.983763 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:38.978935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44932 (local address 127.8.102.1:35343)
0926 15:00:38.979122 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:38.979128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:38.979139 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:00:38.979183 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:38.979185 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:38.979205 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:38.979464 (+ 259us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:38.979527 (+ 63us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.980858 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.982112 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:38.982833 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:38.983310 (+ 477us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:38.983421 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:38.983446 (+ 25us) server_negotiation.cc:299] Negotiation successful
0926 15:00:38.983484 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150,"thread_start_us":83,"threads_started":1}
I20260926 15:00:38.983868 15870 tablet_service.cc:1467] Processing CreateTablet for tablet 42e77da4316343eb9bafc3b0754e6d0e (DEFAULT_TABLE table=test-table [id=15856e4955f34a6287135cf62a44e881]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:38.984238 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e77da4316343eb9bafc3b0754e6d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:38.984310 15734 tablet_service.cc:1467] Processing CreateTablet for tablet 42e77da4316343eb9bafc3b0754e6d0e (DEFAULT_TABLE table=test-table [id=15856e4955f34a6287135cf62a44e881]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:38.984567 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e77da4316343eb9bafc3b0754e6d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:38.987320 15966 tablet_bootstrap.cc:492] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a: Bootstrap starting.
I20260926 15:00:38.987335 15597 tablet_service.cc:1467] Processing CreateTablet for tablet 42e77da4316343eb9bafc3b0754e6d0e (DEFAULT_TABLE table=test-table [id=15856e4955f34a6287135cf62a44e881]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:38.987630 15967 tablet_bootstrap.cc:492] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5: Bootstrap starting.
I20260926 15:00:38.987677 15597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42e77da4316343eb9bafc3b0754e6d0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:38.988101 15966 tablet_bootstrap.cc:654] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:38.988235 15967 tablet_bootstrap.cc:654] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:38.988512 15966 log.cc:824] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:38.988543 15967 log.cc:824] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:38.989415 15966 tablet_bootstrap.cc:492] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a: No bootstrap required, opened a new log
I20260926 15:00:38.989492 15966 ts_tablet_manager.cc:1397] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:00:38.989703 15967 tablet_bootstrap.cc:492] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5: No bootstrap required, opened a new log
I20260926 15:00:38.989770 15967 ts_tablet_manager.cc:1397] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:00:38.990949 15970 tablet_bootstrap.cc:492] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615: Bootstrap starting.
I20260926 15:00:38.991513 15970 tablet_bootstrap.cc:654] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:38.991809 15970 log.cc:824] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:38.991768 15966 raft_consensus.cc:348] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 } } peers { permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 } } peers { permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 } }
I20260926 15:00:38.991768 15967 raft_consensus.cc:348] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 } } peers { permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 } } peers { permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 } }
I20260926 15:00:38.991942 15967 raft_consensus.cc:374] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:38.991942 15966 raft_consensus.cc:374] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:38.991979 15967 raft_consensus.cc:729] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d53cf9c2da74a33bb309bdea1008cd5, State: Initialized, Role: FOLLOWER
I20260926 15:00:38.991979 15966 raft_consensus.cc:729] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e82ec9077f6244fa83a17bfcfe8af69a, State: Initialized, Role: FOLLOWER
I20260926 15:00:38.992110 15966 consensus_queue.cc:260] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 } } peers { permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 } } peers { permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 } }
I20260926 15:00:38.992118 15967 consensus_queue.cc:260] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 } } peers { permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 } } peers { permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 } }
I20260926 15:00:38.992383 15966 ts_tablet_manager.cc:1428] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:00:38.992383 15967 ts_tablet_manager.cc:1428] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:00:38.992524 15801 heartbeater.cc:503] Master 127.8.102.62:44775 was elected leader, sending a full tablet report...
I20260926 15:00:38.992812 15937 heartbeater.cc:503] Master 127.8.102.62:44775 was elected leader, sending a full tablet report...
I20260926 15:00:38.993008 15970 tablet_bootstrap.cc:492] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615: No bootstrap required, opened a new log
I20260926 15:00:38.993091 15970 ts_tablet_manager.cc:1397] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:00:38.995169 15970 raft_consensus.cc:348] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 } } peers { permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 } } peers { permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 } }
I20260926 15:00:38.995301 15970 raft_consensus.cc:374] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:38.995322 15970 raft_consensus.cc:729] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9425f4a721a6474bb40adb111db99615, State: Initialized, Role: FOLLOWER
I20260926 15:00:38.995440 15970 consensus_queue.cc:260] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 } } peers { permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 } } peers { permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 } }
I20260926 15:00:38.995687 15970 ts_tablet_manager.cc:1428] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:00:38.995723 15664 heartbeater.cc:503] Master 127.8.102.62:44775 was elected leader, sending a full tablet report...
W20260926 15:00:39.032472 15803 tablet.cc:2369] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:00:39.139433 15666 tablet.cc:2369] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:00:39.200933 15939 tablet.cc:2369] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:00:39.382851 15974 raft_consensus.cc:482] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:39.383052 15974 raft_consensus.cc:504] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 } } peers { permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 } } peers { permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 } }
I20260926 15:00:39.383486 15974 leader_election.cc:288] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5d53cf9c2da74a33bb309bdea1008cd5 (127.8.102.2:33887), e82ec9077f6244fa83a17bfcfe8af69a (127.8.102.3:36979)
I20260926 15:00:39.387686 15963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:39.383807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:57753 (local address 127.8.102.3:36979)
0926 15:00:39.383857 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:39.383859 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:39.383870 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:00:39.383960 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:39.383961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:39.383973 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:39.384014 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:39.384259 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:39.384992 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:39.386727 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:39.387267 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:39.387344 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:39.387354 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:39.387569 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 15:00:39.387614 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 15:00:39.387686 15962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:39.383736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:58239 (local address 127.8.102.2:33887)
0926 15:00:39.383785 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:39.383788 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:39.383884 (+ 96us) server_negotiation.cc:407] Connection header received
0926 15:00:39.383960 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:39.383961 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:39.383973 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:39.384048 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:39.384205 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:39.384958 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:39.386723 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:39.387269 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:39.387344 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:39.387355 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:39.387569 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 15:00:39.387614 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 15:00:39.387712 15976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:39.383674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:33887 (local address 127.8.102.1:58239)
0926 15:00:39.383800 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:39.383814 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:39.383896 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:39.384063 (+ 167us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:39.384064 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:39.384067 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:39.384130 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:39.384133 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:39.385083 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:39.385084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:39.386609 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:39.386612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:39.387366 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:39.387367 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:39.387452 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:39.387457 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:00:39.387541 (+ 84us) client_negotiation.cc:240] Negotiation successful
0926 15:00:39.387589 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":89,"thread_start_us":67,"threads_started":1}
I20260926 15:00:39.387715 15975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:39.383686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:36979 (local address 127.8.102.1:57753)
0926 15:00:39.383815 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:39.383824 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:39.383880 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:39.384149 (+ 269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:39.384150 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:39.384152 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:39.384186 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:39.384187 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:39.385081 (+ 894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:39.385083 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:39.386609 (+ 1526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:39.386612 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:39.387412 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:39.387412 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:39.387466 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:39.387469 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:00:39.387536 (+ 67us) client_negotiation.cc:240] Negotiation successful
0926 15:00:39.387589 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":60,"thread_start_us":55,"threads_started":1}
I20260926 15:00:39.388063 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e77da4316343eb9bafc3b0754e6d0e" candidate_uuid: "9425f4a721a6474bb40adb111db99615" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" is_pre_election: true
I20260926 15:00:39.388271 15754 raft_consensus.cc:2454] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9425f4a721a6474bb40adb111db99615 in term 0.
I20260926 15:00:39.388315 15890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e77da4316343eb9bafc3b0754e6d0e" candidate_uuid: "9425f4a721a6474bb40adb111db99615" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" is_pre_election: true
I20260926 15:00:39.388523 15890 raft_consensus.cc:2454] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9425f4a721a6474bb40adb111db99615 in term 0.
I20260926 15:00:39.388511 15553 leader_election.cc:302] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d53cf9c2da74a33bb309bdea1008cd5, 9425f4a721a6474bb40adb111db99615; no voters:
I20260926 15:00:39.388648 15974 raft_consensus.cc:2781] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:00:39.388710 15974 raft_consensus.cc:482] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:00:39.388732 15974 raft_consensus.cc:3037] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:39.389425 15974 raft_consensus.cc:504] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 } } peers { permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 } } peers { permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 } }
I20260926 15:00:39.389612 15974 leader_election.cc:288] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [CANDIDATE]: Term 1 election: Requested vote from peers 5d53cf9c2da74a33bb309bdea1008cd5 (127.8.102.2:33887), e82ec9077f6244fa83a17bfcfe8af69a (127.8.102.3:36979)
I20260926 15:00:39.389706 15754 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e77da4316343eb9bafc3b0754e6d0e" candidate_uuid: "9425f4a721a6474bb40adb111db99615" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5d53cf9c2da74a33bb309bdea1008cd5"
I20260926 15:00:39.389782 15754 raft_consensus.cc:3037] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:39.389922 15890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42e77da4316343eb9bafc3b0754e6d0e" candidate_uuid: "9425f4a721a6474bb40adb111db99615" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e82ec9077f6244fa83a17bfcfe8af69a"
I20260926 15:00:39.390053 15890 raft_consensus.cc:3037] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:39.390491 15754 raft_consensus.cc:2454] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9425f4a721a6474bb40adb111db99615 in term 1.
I20260926 15:00:39.390671 15553 leader_election.cc:302] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d53cf9c2da74a33bb309bdea1008cd5, 9425f4a721a6474bb40adb111db99615; no voters:
I20260926 15:00:39.390763 15974 raft_consensus.cc:2781] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:39.390887 15974 raft_consensus.cc:686] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [term 1 LEADER]: Becoming Leader. State: Replica: 9425f4a721a6474bb40adb111db99615, State: Running, Role: LEADER
I20260926 15:00:39.390981 15974 consensus_queue.cc:237] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 } } peers { permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 } } peers { permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 } }
I20260926 15:00:39.391049 15890 raft_consensus.cc:2454] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9425f4a721a6474bb40adb111db99615 in term 1.
I20260926 15:00:39.391853 14699 catalog_manager.cc:5495] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9425f4a721a6474bb40adb111db99615 (127.8.102.1). New cstate: current_term: 1 leader_uuid: "9425f4a721a6474bb40adb111db99615" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 } health_report { overall_health: UNKNOWN } } }
I20260926 15:00:39.422326 15964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:39.419894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44940 (local address 127.8.102.1:35343)
0926 15:00:39.419957 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:39.419959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:39.419968 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:39.420015 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:39.420016 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:39.420052 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:39.420130 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:39.420288 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:39.420758 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:39.421506 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:39.421729 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:39.421813 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:39.421993 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:39.422103 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:39.422197 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:39.422215 (+ 18us) server_negotiation.cc:299] Negotiation successful
0926 15:00:39.422236 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 15:00:39.424319 15890 raft_consensus.cc:1261] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a [term 1 FOLLOWER]: Refusing update from remote peer 9425f4a721a6474bb40adb111db99615: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:39.424319 15754 raft_consensus.cc:1261] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5 [term 1 FOLLOWER]: Refusing update from remote peer 9425f4a721a6474bb40adb111db99615: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:39.424772 15977 consensus_queue.cc:1035] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:39.424966 15974 consensus_queue.cc:1035] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:39.427940 15981 mvcc.cc:205] Tried to move back new op lower bound from 7333621102279110656 to 7333621102146011136. Current Snapshot: MvccSnapshot[applied={T|T < 7333621102279110656}]
I20260926 15:00:39.429915 15982 mvcc.cc:205] Tried to move back new op lower bound from 7333621102279110656 to 7333621102146011136. Current Snapshot: MvccSnapshot[applied={T|T < 7333621102279110656}]
I20260926 15:00:39.431320 14700 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:39.431417 14700 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37260:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:39.432868 14700 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 353bc184be504361bc9d470653da6fdc: Sending DeleteTablet for 3 replicas of tablet 42e77da4316343eb9bafc3b0754e6d0e
I20260926 15:00:39.433238 15597 tablet_service.cc:1514] Processing DeleteTablet for tablet 42e77da4316343eb9bafc3b0754e6d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:39 UTC) from {username='oryx', principal='oryx/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:44932
I20260926 15:00:39.433236 15870 tablet_service.cc:1514] Processing DeleteTablet for tablet 42e77da4316343eb9bafc3b0754e6d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:39 UTC) from {username='oryx', principal='oryx/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:38776
I20260926 15:00:39.433547 15990 tablet_replica.cc:330] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a: stopping tablet replica
I20260926 15:00:39.433717 15990 raft_consensus.cc:2229] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:39.433722 15734 tablet_service.cc:1514] Processing DeleteTablet for tablet 42e77da4316343eb9bafc3b0754e6d0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:39 UTC) from {username='oryx', principal='oryx/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:38052
I20260926 15:00:39.433934 15990 raft_consensus.cc:2258] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:39.433962 15989 tablet_replica.cc:330] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615: stopping tablet replica
I20260926 15:00:39.434052 15989 raft_consensus.cc:2229] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:00:39.434242 15989 raft_consensus.cc:2258] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:39.434778 15991 tablet_replica.cc:330] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5: stopping tablet replica
I20260926 15:00:39.434862 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 14668
I20260926 15:00:39.434887 15991 raft_consensus.cc:2229] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:39.436281 15991 raft_consensus.cc:2258] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:39.436781 15991 ts_tablet_manager.cc:1905] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:39.438905 15991 ts_tablet_manager.cc:1918] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:00:39.438983 15991 log.cc:1196] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/42e77da4316343eb9bafc3b0754e6d0e
I20260926 15:00:39.439265 15991 ts_tablet_manager.cc:1939] T 42e77da4316343eb9bafc3b0754e6d0e P 5d53cf9c2da74a33bb309bdea1008cd5: Deleting consensus metadata
W20260926 15:00:39.442373 15688 connection.cc:535] server connection from 127.0.0.1:38052 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 15:00:39.443827 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:39.434329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45260 (local address 127.8.102.60:45261)
0926 15:00:39.434385 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:39.434387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:39.434832 (+ 445us) server_negotiation.cc:407] Connection header received
0926 15:00:39.434857 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:39.434858 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:39.434871 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:39.434925 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:39.436179 (+ 1254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:39.436960 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:39.438035 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:39.438334 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:39.438377 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:39.440195 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:39.440197 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:39.440197 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:39.440205 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:39.442145 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:39.442781 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:39.442783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:39.442783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:39.442850 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:39.443136 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:39.443137 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:39.443138 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:39.443338 (+ 200us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:39.443429 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:39.443644 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 15:00:39.443741 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 15:00:39.444380 15990 ts_tablet_manager.cc:1905] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:39.443831 15989 ts_tablet_manager.cc:1905] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:39.446996 15989 ts_tablet_manager.cc:1918] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:00:39.448318 15990 ts_tablet_manager.cc:1918] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:00:39.448475 15990 log.cc:1196] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/42e77da4316343eb9bafc3b0754e6d0e
I20260926 15:00:39.448578 15989 log.cc:1196] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-0/wal/wals/42e77da4316343eb9bafc3b0754e6d0e
I20260926 15:00:39.449195 15989 ts_tablet_manager.cc:1939] T 42e77da4316343eb9bafc3b0754e6d0e P 9425f4a721a6474bb40adb111db99615: Deleting consensus metadata
W20260926 15:00:39.449630 15551 connection.cc:441] server connection from 127.0.0.1:44932 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44932 (request call id 1) could send its response
I20260926 15:00:39.449811 15990 ts_tablet_manager.cc:1939] T 42e77da4316343eb9bafc3b0754e6d0e P e82ec9077f6244fa83a17bfcfe8af69a: Deleting consensus metadata
W20260926 15:00:39.450242 15826 connection.cc:441] server connection from 127.0.0.1:38776 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:38776 (request call id 1) could send its response
I20260926 15:00:39.453617 15951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:39.444676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49862 (local address 127.8.102.61:34051)
0926 15:00:39.444721 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:39.444723 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:39.444956 (+ 233us) server_negotiation.cc:407] Connection header received
0926 15:00:39.445082 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:39.445092 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:39.445108 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:39.445170 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:39.445922 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:39.446721 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:39.448409 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:39.448732 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:39.448783 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:39.450594 (+ 1811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:39.450595 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:39.450596 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:39.450603 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:39.452381 (+ 1778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:39.452763 (+ 382us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:39.452765 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:39.452765 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:39.452820 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:39.453015 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:39.453016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:39.453016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:39.453171 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:39.453275 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:39.453414 (+ 139us) server_negotiation.cc:299] Negotiation successful
0926 15:00:39.453500 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 15:00:39.455137 15953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:39.445209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45274 (local address 127.8.102.60:45261)
0926 15:00:39.445242 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:39.445244 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:39.446799 (+ 1555us) server_negotiation.cc:407] Connection header received
0926 15:00:39.446914 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:39.446915 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:39.446924 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:39.446952 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:39.447734 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:39.448535 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:39.450080 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:39.450482 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:39.450536 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:39.452114 (+ 1578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:39.452116 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:39.452116 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:39.452124 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:39.453918 (+ 1794us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:39.454348 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:39.454349 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:39.454349 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:39.454408 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:39.454608 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:39.454608 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:39.454609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:39.454782 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:39.454861 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:39.454997 (+ 136us) server_negotiation.cc:299] Negotiation successful
0926 15:00:39.455068 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 15:00:40.428884 15999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:40.428591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.3:39867)
0926 15:00:40.428720 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:40.428817 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":84,"threads_started":1}
W20260926 15:00:40.429101 15937 heartbeater.cc:644] Failed to heartbeat to 127.8.102.62:44775 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
I20260926 15:00:40.430279 16000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:40.430035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.1:46731)
0926 15:00:40.430153 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:40.430234 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
W20260926 15:00:40.430537 15664 heartbeater.cc:644] Failed to heartbeat to 127.8.102.62:44775 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
I20260926 15:00:40.431037 16001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:40.430826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.2:58677)
0926 15:00:40.430927 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:40.430996 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":64,"threads_started":1}
W20260926 15:00:40.431212 15801 heartbeater.cc:644] Failed to heartbeat to 127.8.102.62:44775 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
I20260926 15:00:40.933799 16002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 353bc184be504361bc9d470653da6fdc)
I20260926 15:00:40.933952 16003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 353bc184be504361bc9d470653da6fdc)
I20260926 15:00:40.933952 16002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false } }
I20260926 15:00:40.934105 16003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false } }
I20260926 15:00:40.934254 16002 leader_election.cc:288] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 353bc184be504361bc9d470653da6fdc (127.8.102.62:44775), 26a22fd6ed974376bb5a6d4da9a14957 (127.8.102.61:34051)
I20260926 15:00:40.934369 16003 leader_election.cc:288] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 353bc184be504361bc9d470653da6fdc (127.8.102.62:44775), 98eaf0f289e747c7bc3c658f5fd5a84d (127.8.102.60:45261)
I20260926 15:00:40.934597 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:40.934404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.0.0.1:37718)
0926 15:00:40.934495 (+ 91us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:40.934548 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71,"thread_start_us":44,"threads_started":1}
W20260926 15:00:40.934926 15471 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
I20260926 15:00:40.934962 16006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:40.934486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.0.0.1:37732)
0926 15:00:40.934611 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:40.934914 (+ 303us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":52,"threads_started":1}
W20260926 15:00:40.935225 15284 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
I20260926 15:00:40.935446 16004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:40.935333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.0.0.1:37738)
0926 15:00:40.935396 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:40.935422 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260926 15:00:40.935524 15471 leader_election.cc:334] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 353bc184be504361bc9d470653da6fdc (127.8.102.62:44775): Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
I20260926 15:00:40.935747 16006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:40.935654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.0.0.1:37750)
0926 15:00:40.935691 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:40.935722 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
W20260926 15:00:40.935900 15284 leader_election.cc:334] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 353bc184be504361bc9d470653da6fdc (127.8.102.62:44775): Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
I20260926 15:00:40.937763 16008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:40.934761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49872 (local address 127.8.102.61:34051)
0926 15:00:40.934872 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:40.934874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:40.934881 (+ 7us) server_negotiation.cc:407] Connection header received
0926 15:00:40.934903 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:40.934903 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:40.934912 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:40.935158 (+ 246us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:40.935193 (+ 35us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:40.936033 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:40.937022 (+ 989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:40.937403 (+ 381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:40.937502 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:40.937515 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:40.937652 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 15:00:40.937692 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96,"thread_start_us":41,"threads_started":1}
I20260926 15:00:40.937767 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:40.934555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:34051 (local address 127.0.0.1:49872)
0926 15:00:40.934692 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:40.934703 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:40.934767 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:40.935029 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:40.935031 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:40.935033 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:40.935097 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:40.935099 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:40.936114 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:40.936115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:40.936893 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:40.936895 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:40.937505 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:40.937506 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:40.937578 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:40.937582 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:00:40.937648 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 15:00:40.937695 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":51,"threads_started":1}
I20260926 15:00:40.938059 15307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "26a22fd6ed974376bb5a6d4da9a14957" is_pre_election: true
I20260926 15:00:40.938154 16005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:40.934489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:45261 (local address 127.0.0.1:45276)
0926 15:00:40.934774 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:40.934784 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:40.934830 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:40.935055 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:40.935055 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:40.935058 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:40.935115 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:40.935117 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:40.936305 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:40.936306 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:40.937215 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:40.937218 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:40.937917 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:40.937918 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:40.937981 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:40.937984 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:00:40.938019 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 15:00:40.938061 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":267,"thread_start_us":45,"threads_started":1}
I20260926 15:00:40.938210 16009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:40.934770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45276 (local address 127.8.102.60:45261)
0926 15:00:40.934863 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:40.934865 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:40.934874 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:40.934906 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:40.934907 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:40.934918 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:40.935000 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:40.935184 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:40.936197 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:40.937339 (+ 1142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:40.937758 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:40.938047 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:40.938062 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:40.938082 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 15:00:40.938128 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":43,"threads_started":1}
I20260926 15:00:40.938221 15307 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 98eaf0f289e747c7bc3c658f5fd5a84d in term 1.
I20260926 15:00:40.938391 15471 leader_election.cc:302] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26a22fd6ed974376bb5a6d4da9a14957, 98eaf0f289e747c7bc3c658f5fd5a84d; no voters: 353bc184be504361bc9d470653da6fdc
I20260926 15:00:40.938448 15494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26a22fd6ed974376bb5a6d4da9a14957" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" is_pre_election: true
I20260926 15:00:40.938527 16002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 15:00:40.938603 15494 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26a22fd6ed974376bb5a6d4da9a14957 in term 1.
I20260926 15:00:40.938663 16002 raft_consensus.cc:482] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 353bc184be504361bc9d470653da6fdc)
I20260926 15:00:40.938704 16002 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 1 FOLLOWER]: Advancing to term 2
I20260926 15:00:40.938788 15281 leader_election.cc:302] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26a22fd6ed974376bb5a6d4da9a14957, 98eaf0f289e747c7bc3c658f5fd5a84d; no voters: 353bc184be504361bc9d470653da6fdc
I20260926 15:00:40.938887 16003 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260926 15:00:40.938939 16003 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 353bc184be504361bc9d470653da6fdc)
I20260926 15:00:40.938970 16003 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 1 FOLLOWER]: Advancing to term 2
I20260926 15:00:40.939730 16002 raft_consensus.cc:504] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false } }
I20260926 15:00:40.939920 16002 leader_election.cc:288] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [CANDIDATE]: Term 2 election: Requested vote from peers 353bc184be504361bc9d470653da6fdc (127.8.102.62:44775), 26a22fd6ed974376bb5a6d4da9a14957 (127.8.102.61:34051)
I20260926 15:00:40.939890 16003 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false } }
I20260926 15:00:40.940042 16003 leader_election.cc:288] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [CANDIDATE]: Term 2 election: Requested vote from peers 353bc184be504361bc9d470653da6fdc (127.8.102.62:44775), 98eaf0f289e747c7bc3c658f5fd5a84d (127.8.102.60:45261)
I20260926 15:00:40.940166 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:40.940018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.0.0.1:37764)
0926 15:00:40.940065 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:40.940135 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260926 15:00:40.940210 16005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:40.940127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.0.0.1:37768)
0926 15:00:40.940158 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:40.940185 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 15:00:40.940212 15307 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "26a22fd6ed974376bb5a6d4da9a14957"
I20260926 15:00:40.940339 15307 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 98eaf0f289e747c7bc3c658f5fd5a84d in current term 2: Already voted for candidate 26a22fd6ed974376bb5a6d4da9a14957 in this term.
I20260926 15:00:40.940340 15494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26a22fd6ed974376bb5a6d4da9a14957" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d"
I20260926 15:00:40.940428 15494 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 26a22fd6ed974376bb5a6d4da9a14957 in current term 2: Already voted for candidate 98eaf0f289e747c7bc3c658f5fd5a84d in this term.
I20260926 15:00:40.940496 16007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:40.940392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.0.0.1:37770)
0926 15:00:40.940449 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:40.940473 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 15:00:40.940543 15471 leader_election.cc:334] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 353bc184be504361bc9d470653da6fdc (127.8.102.62:44775): Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
I20260926 15:00:40.940686 15471 leader_election.cc:302] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 98eaf0f289e747c7bc3c658f5fd5a84d; no voters: 26a22fd6ed974376bb5a6d4da9a14957, 353bc184be504361bc9d470653da6fdc
I20260926 15:00:40.940743 16005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:40.940575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.0.0.1:37784)
0926 15:00:40.940707 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:40.940727 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260926 15:00:40.940845 15284 leader_election.cc:334] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 353bc184be504361bc9d470653da6fdc (127.8.102.62:44775): Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
I20260926 15:00:40.940899 15284 leader_election.cc:302] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26a22fd6ed974376bb5a6d4da9a14957; no voters: 353bc184be504361bc9d470653da6fdc, 98eaf0f289e747c7bc3c658f5fd5a84d
I20260926 15:00:40.940972 16003 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 15:00:40.941028 16002 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20260926 15:00:41.429737 16012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:41.429468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.3:53287)
0926 15:00:41.429630 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:41.429681 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":81,"threads_started":1}
I20260926 15:00:41.431102 16013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:41.430867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.1:54013)
0926 15:00:41.431008 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:41.431052 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260926 15:00:41.431799 16014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:41.431567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.2:38787)
0926 15:00:41.431718 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:41.431755 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":72,"threads_started":1}
I20260926 15:00:42.430434 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.430211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.3:35213)
0926 15:00:42.430333 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.430376 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":73,"threads_started":1}
I20260926 15:00:42.431663 16016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.431446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.1:40367)
0926 15:00:42.431578 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.431617 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
I20260926 15:00:42.432420 16017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.432175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.2:42763)
0926 15:00:42.432329 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.432365 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":50,"threads_started":1}
I20260926 15:00:42.481503 16018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:42.481599 16018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false } }
I20260926 15:00:42.481781 16018 leader_election.cc:288] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 353bc184be504361bc9d470653da6fdc (127.8.102.62:44775), 98eaf0f289e747c7bc3c658f5fd5a84d (127.8.102.60:45261)
I20260926 15:00:42.482055 15494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26a22fd6ed974376bb5a6d4da9a14957" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" is_pre_election: true
I20260926 15:00:42.482141 15494 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26a22fd6ed974376bb5a6d4da9a14957 in term 2.
I20260926 15:00:42.482160 16019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.481923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.0.0.1:50048)
0926 15:00:42.482061 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.482103 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260926 15:00:42.482292 15281 leader_election.cc:302] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26a22fd6ed974376bb5a6d4da9a14957, 98eaf0f289e747c7bc3c658f5fd5a84d; no voters:
I20260926 15:00:42.482391 16018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260926 15:00:42.482425 16018 raft_consensus.cc:482] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:00:42.482440 16018 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 2 FOLLOWER]: Advancing to term 3
I20260926 15:00:42.482791 16019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.482684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.0.0.1:50064)
0926 15:00:42.482726 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.482757 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260926 15:00:42.482910 15284 leader_election.cc:334] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 353bc184be504361bc9d470653da6fdc (127.8.102.62:44775): Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
I20260926 15:00:42.483309 16018 raft_consensus.cc:504] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false } }
I20260926 15:00:42.483443 16018 leader_election.cc:288] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [CANDIDATE]: Term 3 election: Requested vote from peers 353bc184be504361bc9d470653da6fdc (127.8.102.62:44775), 98eaf0f289e747c7bc3c658f5fd5a84d (127.8.102.60:45261)
I20260926 15:00:42.483656 15494 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "26a22fd6ed974376bb5a6d4da9a14957" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d"
I20260926 15:00:42.483692 16019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.483517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.0.0.1:50066)
0926 15:00:42.483634 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.483670 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260926 15:00:42.483743 15494 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 2 FOLLOWER]: Advancing to term 3
I20260926 15:00:42.483999 16019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.483911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.0.0.1:50070)
0926 15:00:42.483954 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.483979 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260926 15:00:42.484074 15284 leader_election.cc:334] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer 353bc184be504361bc9d470653da6fdc (127.8.102.62:44775): Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
I20260926 15:00:42.484478 15494 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26a22fd6ed974376bb5a6d4da9a14957 in term 3.
I20260926 15:00:42.484655 15281 leader_election.cc:302] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 26a22fd6ed974376bb5a6d4da9a14957, 98eaf0f289e747c7bc3c658f5fd5a84d; no voters: 353bc184be504361bc9d470653da6fdc
I20260926 15:00:42.484745 16018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 15:00:42.484854 16018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [term 3 LEADER]: Becoming Leader. State: Replica: 26a22fd6ed974376bb5a6d4da9a14957, State: Running, Role: LEADER
I20260926 15:00:42.484911 16018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false } }
I20260926 15:00:42.485226 16021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26a22fd6ed974376bb5a6d4da9a14957. Latest consensus state: current_term: 3 leader_uuid: "26a22fd6ed974376bb5a6d4da9a14957" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false } } }
I20260926 15:00:42.485296 16021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:42.485457 16023 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:00:42.485919 16023 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:00:42.486038 16023 catalog_manager.cc:1259] Loaded cluster ID: 509e0d816e344dd3ab0df33248b2f26b
I20260926 15:00:42.486074 16023 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:00:42.486267 16023 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:00:42.486364 16023 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: Loaded TSK: 0
I20260926 15:00:42.486518 16023 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:00:42.550786 15296 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49862:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:00:42.552486 15296 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:00:42.553515 16019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.553399 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.0.0.1:50086)
0926 15:00:42.553451 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.553483 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260926 15:00:42.553577 15494 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [term 3 FOLLOWER]: Refusing update from remote peer 26a22fd6ed974376bb5a6d4da9a14957: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20260926 15:00:42.553862 16019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.553740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.0.0.1:50090)
0926 15:00:42.553776 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.553810 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260926 15:00:42.553808 16018 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [LEADER]: Connected to new peer: Peer: permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260926 15:00:42.553947 15284 consensus_peers.cc:487] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 -> Peer 353bc184be504361bc9d470653da6fdc (127.8.102.62:44775): Couldn't send request to peer 353bc184be504361bc9d470653da6fdc. Status: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260926 15:00:42.554416 16025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26a22fd6ed974376bb5a6d4da9a14957. Latest consensus state: current_term: 3 leader_uuid: "26a22fd6ed974376bb5a6d4da9a14957" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false } } }
I20260926 15:00:42.554525 16025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:00:42.555027 16018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "26a22fd6ed974376bb5a6d4da9a14957" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false } } }
I20260926 15:00:42.555133 16018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:42.555194 16025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "26a22fd6ed974376bb5a6d4da9a14957" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "353bc184be504361bc9d470653da6fdc" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44775 } } peers { permanent_uuid: "26a22fd6ed974376bb5a6d4da9a14957" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34051 } attrs { promote: false } } peers { permanent_uuid: "98eaf0f289e747c7bc3c658f5fd5a84d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 45261 } attrs { promote: false } } }
I20260926 15:00:42.555266 16025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 98eaf0f289e747c7bc3c658f5fd5a84d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:00:42.560101 16019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.556743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:36979 (local address 127.0.0.1:50432)
0926 15:00:42.556891 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.556903 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:42.556956 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:42.557137 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:42.557138 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:42.557140 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:42.557195 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:42.557197 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.558025 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.558026 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:42.558897 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:42.558899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.559753 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.559754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:42.559914 (+ 160us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:42.559918 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:00:42.559977 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 15:00:42.560025 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"mutex_wait_us":3}
I20260926 15:00:42.560114 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.556794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50432 (local address 127.8.102.3:36979)
0926 15:00:42.556926 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:42.556928 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:42.556940 (+ 12us) server_negotiation.cc:407] Connection header received
0926 15:00:42.557039 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:42.557040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:42.557050 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:42.557137 (+ 87us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:42.557260 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.557941 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.558979 (+ 1038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.559603 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.559694 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:42.559707 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:42.559995 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 15:00:42.560040 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108,"thread_start_us":51,"threads_started":1}
I20260926 15:00:42.560631 16033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.556803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:33887 (local address 127.0.0.1:34656)
0926 15:00:42.557228 (+ 425us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.557236 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:42.557276 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:42.557413 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:42.557414 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:42.557415 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:42.557464 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:42.557466 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.558383 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.558383 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:42.559622 (+ 1239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:42.559624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.560396 (+ 772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.560396 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:42.560438 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:42.560441 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:00:42.560507 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 15:00:42.560548 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"thread_start_us":59,"threads_started":1}
I20260926 15:00:42.560642 16034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.556864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34656 (local address 127.8.102.2:33887)
0926 15:00:42.556996 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:42.556998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:42.557319 (+ 321us) server_negotiation.cc:407] Connection header received
0926 15:00:42.557348 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:42.557349 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:42.557359 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:42.557414 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:42.557548 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.558286 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.559716 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.560302 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.560341 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:42.560350 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:42.560511 (+ 161us) server_negotiation.cc:299] Negotiation successful
0926 15:00:42.560553 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":52,"threads_started":1}
I20260926 15:00:42.560868 15870 tablet_service.cc:1467] Processing CreateTablet for tablet 91c0c0773cc54400a4d4661d74b2ff4d (DEFAULT_TABLE table=test-table [id=22c17930aeb141359f88f564f26f1a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:42.560948 16032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.556777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:35343 (local address 127.0.0.1:41800)
0926 15:00:42.557108 (+ 331us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.557118 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:42.557157 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:42.557646 (+ 489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:42.557647 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:42.557649 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:42.557687 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:42.557689 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.558539 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.558540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:42.559722 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:42.559723 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.560691 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.560693 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:42.560771 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:42.560775 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:00:42.560830 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 15:00:42.560870 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":315,"mutex_wait_us":22,"thread_start_us":76,"threads_started":1}
I20260926 15:00:42.561019 15870 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c0c0773cc54400a4d4661d74b2ff4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:42.561198 16035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.557103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41800 (local address 127.8.102.1:35343)
0926 15:00:42.557523 (+ 420us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:42.557525 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:42.557532 (+ 7us) server_negotiation.cc:407] Connection header received
0926 15:00:42.557556 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:42.557557 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:42.557569 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:42.557632 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:42.557759 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.558448 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.559797 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.560463 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.560624 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:42.560645 (+ 21us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:42.561054 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 15:00:42.561120 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":62,"threads_started":1}
I20260926 15:00:42.562466 16036 tablet_bootstrap.cc:492] T 91c0c0773cc54400a4d4661d74b2ff4d P e82ec9077f6244fa83a17bfcfe8af69a: Bootstrap starting.
I20260926 15:00:42.562695 15597 tablet_service.cc:1467] Processing CreateTablet for tablet 91c0c0773cc54400a4d4661d74b2ff4d (DEFAULT_TABLE table=test-table [id=22c17930aeb141359f88f564f26f1a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:42.562813 15597 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c0c0773cc54400a4d4661d74b2ff4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:42.562920 16036 tablet_bootstrap.cc:654] T 91c0c0773cc54400a4d4661d74b2ff4d P e82ec9077f6244fa83a17bfcfe8af69a: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:42.563550 16036 tablet_bootstrap.cc:492] T 91c0c0773cc54400a4d4661d74b2ff4d P e82ec9077f6244fa83a17bfcfe8af69a: No bootstrap required, opened a new log
I20260926 15:00:42.563596 16036 ts_tablet_manager.cc:1397] T 91c0c0773cc54400a4d4661d74b2ff4d P e82ec9077f6244fa83a17bfcfe8af69a: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:42.563735 16036 raft_consensus.cc:348] T 91c0c0773cc54400a4d4661d74b2ff4d P e82ec9077f6244fa83a17bfcfe8af69a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 } } peers { permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 } } peers { permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 } }
I20260926 15:00:42.563804 16036 raft_consensus.cc:374] T 91c0c0773cc54400a4d4661d74b2ff4d P e82ec9077f6244fa83a17bfcfe8af69a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:42.563819 16036 raft_consensus.cc:729] T 91c0c0773cc54400a4d4661d74b2ff4d P e82ec9077f6244fa83a17bfcfe8af69a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e82ec9077f6244fa83a17bfcfe8af69a, State: Initialized, Role: FOLLOWER
I20260926 15:00:42.563872 16036 consensus_queue.cc:260] T 91c0c0773cc54400a4d4661d74b2ff4d P e82ec9077f6244fa83a17bfcfe8af69a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 } } peers { permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 } } peers { permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 } }
I20260926 15:00:42.564050 16036 ts_tablet_manager.cc:1428] T 91c0c0773cc54400a4d4661d74b2ff4d P e82ec9077f6244fa83a17bfcfe8af69a: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260926 15:00:42.564355 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.564248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.3:55027)
0926 15:00:42.564285 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.564326 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
I20260926 15:00:42.564388 15734 tablet_service.cc:1467] Processing CreateTablet for tablet 91c0c0773cc54400a4d4661d74b2ff4d (DEFAULT_TABLE table=test-table [id=22c17930aeb141359f88f564f26f1a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:42.564508 15734 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 91c0c0773cc54400a4d4661d74b2ff4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:42.564532 16039 tablet_bootstrap.cc:492] T 91c0c0773cc54400a4d4661d74b2ff4d P 9425f4a721a6474bb40adb111db99615: Bootstrap starting.
I20260926 15:00:42.564683 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.564602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.3:39785)
0926 15:00:42.564635 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.564664 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
I20260926 15:00:42.565035 16039 tablet_bootstrap.cc:654] T 91c0c0773cc54400a4d4661d74b2ff4d P 9425f4a721a6474bb40adb111db99615: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:42.565604 16039 tablet_bootstrap.cc:492] T 91c0c0773cc54400a4d4661d74b2ff4d P 9425f4a721a6474bb40adb111db99615: No bootstrap required, opened a new log
I20260926 15:00:42.565649 16039 ts_tablet_manager.cc:1397] T 91c0c0773cc54400a4d4661d74b2ff4d P 9425f4a721a6474bb40adb111db99615: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:42.565754 16039 raft_consensus.cc:348] T 91c0c0773cc54400a4d4661d74b2ff4d P 9425f4a721a6474bb40adb111db99615 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 } } peers { permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 } } peers { permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 } }
I20260926 15:00:42.565825 16039 raft_consensus.cc:374] T 91c0c0773cc54400a4d4661d74b2ff4d P 9425f4a721a6474bb40adb111db99615 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:42.565850 16039 raft_consensus.cc:729] T 91c0c0773cc54400a4d4661d74b2ff4d P 9425f4a721a6474bb40adb111db99615 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9425f4a721a6474bb40adb111db99615, State: Initialized, Role: FOLLOWER
I20260926 15:00:42.565907 16039 consensus_queue.cc:260] T 91c0c0773cc54400a4d4661d74b2ff4d P 9425f4a721a6474bb40adb111db99615 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 } } peers { permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 } } peers { permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 } }
I20260926 15:00:42.566040 16041 tablet_bootstrap.cc:492] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5: Bootstrap starting.
I20260926 15:00:42.566092 16039 ts_tablet_manager.cc:1428] T 91c0c0773cc54400a4d4661d74b2ff4d P 9425f4a721a6474bb40adb111db99615: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 15:00:42.566419 16016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.566258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.1:35547)
0926 15:00:42.566361 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.566388 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260926 15:00:42.566838 16016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.566611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.1:45803)
0926 15:00:42.566783 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.566812 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20260926 15:00:42.566864 15938 heartbeater.cc:503] Master 127.8.102.61:34051 was elected leader, sending a full tablet report...
I20260926 15:00:42.567266 15665 heartbeater.cc:503] Master 127.8.102.61:34051 was elected leader, sending a full tablet report...
I20260926 15:00:42.567323 16041 tablet_bootstrap.cc:654] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:42.568126 16041 tablet_bootstrap.cc:492] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5: No bootstrap required, opened a new log
I20260926 15:00:42.568172 16041 ts_tablet_manager.cc:1397] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:00:42.568329 16041 raft_consensus.cc:348] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 } } peers { permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 } } peers { permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 } }
I20260926 15:00:42.568384 16041 raft_consensus.cc:374] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:42.568398 16041 raft_consensus.cc:729] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5d53cf9c2da74a33bb309bdea1008cd5, State: Initialized, Role: FOLLOWER
I20260926 15:00:42.568467 16041 consensus_queue.cc:260] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 } } peers { permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 } } peers { permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 } }
I20260926 15:00:42.568656 16041 ts_tablet_manager.cc:1428] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 15:00:42.568905 16017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.568802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.2:51111)
0926 15:00:42.568846 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.568875 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260926 15:00:42.569454 15802 heartbeater.cc:503] Master 127.8.102.61:34051 was elected leader, sending a full tablet report...
I20260926 15:00:42.569525 16017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.569380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.2:35213)
0926 15:00:42.569435 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.569468 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 15:00:42.700049 16044 raft_consensus.cc:482] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:42.700165 16044 raft_consensus.cc:504] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 } } peers { permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 } } peers { permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 } }
I20260926 15:00:42.700529 16044 leader_election.cc:288] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e82ec9077f6244fa83a17bfcfe8af69a (127.8.102.3:36979), 9425f4a721a6474bb40adb111db99615 (127.8.102.1:35343)
I20260926 15:00:42.705143 16031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.700748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:44911 (local address 127.8.102.3:36979)
0926 15:00:42.700804 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:42.700807 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:42.700976 (+ 169us) server_negotiation.cc:407] Connection header received
0926 15:00:42.701021 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:42.701022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:42.701035 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:42.701090 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:42.701284 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.702034 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.704037 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.704748 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.704818 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:42.704831 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:42.705032 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 15:00:42.705075 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 15:00:42.705144 16035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.700774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:55821 (local address 127.8.102.1:35343)
0926 15:00:42.700821 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:42.700824 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:42.700881 (+ 57us) server_negotiation.cc:407] Connection header received
0926 15:00:42.700939 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:42.700940 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:42.700976 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:42.701078 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:42.701286 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.702036 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.704036 (+ 2000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.704676 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.704773 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:42.704802 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:42.705007 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 15:00:42.705059 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 15:00:42.705204 16017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.700706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:35343 (local address 127.8.102.2:55821)
0926 15:00:42.700792 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.700805 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:42.700877 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:42.701092 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:42.701092 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:42.701096 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:42.701208 (+ 112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:42.701210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.702157 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.702159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:42.703892 (+ 1733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:42.703896 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.704804 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.704806 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:42.704897 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:42.704902 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:00:42.704976 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 15:00:42.705035 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67,"mutex_wait_us":166}
I20260926 15:00:42.705212 16045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.700726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:36979 (local address 127.8.102.2:44911)
0926 15:00:42.700903 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.700914 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:42.700970 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:42.701106 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:42.701107 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:42.701108 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:42.701208 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:42.701210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.702157 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.702159 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:42.703891 (+ 1732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:42.703895 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.704856 (+ 961us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.704857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:42.704952 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:42.704956 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:00:42.705015 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 15:00:42.705050 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":116,"thread_start_us":102,"threads_started":1}
I20260926 15:00:42.705551 15617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c0c0773cc54400a4d4661d74b2ff4d" candidate_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9425f4a721a6474bb40adb111db99615" is_pre_election: true
I20260926 15:00:42.705610 15890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c0c0773cc54400a4d4661d74b2ff4d" candidate_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" is_pre_election: true
I20260926 15:00:42.705713 15890 raft_consensus.cc:2454] T 91c0c0773cc54400a4d4661d74b2ff4d P e82ec9077f6244fa83a17bfcfe8af69a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d53cf9c2da74a33bb309bdea1008cd5 in term 0.
I20260926 15:00:42.705736 15617 raft_consensus.cc:2454] T 91c0c0773cc54400a4d4661d74b2ff4d P 9425f4a721a6474bb40adb111db99615 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5d53cf9c2da74a33bb309bdea1008cd5 in term 0.
I20260926 15:00:42.705878 15688 leader_election.cc:302] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d53cf9c2da74a33bb309bdea1008cd5, e82ec9077f6244fa83a17bfcfe8af69a; no voters:
I20260926 15:00:42.706060 16044 raft_consensus.cc:2781] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:00:42.706108 16044 raft_consensus.cc:482] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:00:42.706125 16044 raft_consensus.cc:3037] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:42.706988 16044 raft_consensus.cc:504] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 } } peers { permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 } } peers { permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 } }
I20260926 15:00:42.707149 16044 leader_election.cc:288] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [CANDIDATE]: Term 1 election: Requested vote from peers e82ec9077f6244fa83a17bfcfe8af69a (127.8.102.3:36979), 9425f4a721a6474bb40adb111db99615 (127.8.102.1:35343)
I20260926 15:00:42.707329 15617 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c0c0773cc54400a4d4661d74b2ff4d" candidate_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9425f4a721a6474bb40adb111db99615"
I20260926 15:00:42.707392 15890 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "91c0c0773cc54400a4d4661d74b2ff4d" candidate_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e82ec9077f6244fa83a17bfcfe8af69a"
I20260926 15:00:42.707440 15617 raft_consensus.cc:3037] T 91c0c0773cc54400a4d4661d74b2ff4d P 9425f4a721a6474bb40adb111db99615 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:42.707464 15890 raft_consensus.cc:3037] T 91c0c0773cc54400a4d4661d74b2ff4d P e82ec9077f6244fa83a17bfcfe8af69a [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:42.708287 15890 raft_consensus.cc:2454] T 91c0c0773cc54400a4d4661d74b2ff4d P e82ec9077f6244fa83a17bfcfe8af69a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d53cf9c2da74a33bb309bdea1008cd5 in term 1.
I20260926 15:00:42.708343 15617 raft_consensus.cc:2454] T 91c0c0773cc54400a4d4661d74b2ff4d P 9425f4a721a6474bb40adb111db99615 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5d53cf9c2da74a33bb309bdea1008cd5 in term 1.
I20260926 15:00:42.708535 15687 leader_election.cc:302] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5d53cf9c2da74a33bb309bdea1008cd5, 9425f4a721a6474bb40adb111db99615; no voters:
I20260926 15:00:42.708719 16044 raft_consensus.cc:2781] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:42.708932 16044 raft_consensus.cc:686] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [term 1 LEADER]: Becoming Leader. State: Replica: 5d53cf9c2da74a33bb309bdea1008cd5, State: Running, Role: LEADER
I20260926 15:00:42.709015 16044 consensus_queue.cc:237] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 } } peers { permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 } } peers { permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 } }
I20260926 15:00:42.709283 16045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.709145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.2:35625)
0926 15:00:42.709201 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.709243 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 15:00:42.709828 15296 catalog_manager.cc:5495] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5d53cf9c2da74a33bb309bdea1008cd5 (127.8.102.2). New cstate: current_term: 1 leader_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5d53cf9c2da74a33bb309bdea1008cd5" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33887 } health_report { overall_health: HEALTHY } } }
I20260926 15:00:42.744045 16034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.741640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34666 (local address 127.8.102.2:33887)
0926 15:00:42.741698 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:42.741700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:42.741724 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:00:42.741766 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:42.741767 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:42.741785 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:42.741839 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:42.741924 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.742457 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.743306 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:42.743534 (+ 228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:42.743594 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:42.743716 (+ 122us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:42.743814 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:42.743878 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:42.743923 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 15:00:42.743942 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 15:00:42.745195 15890 raft_consensus.cc:1261] T 91c0c0773cc54400a4d4661d74b2ff4d P e82ec9077f6244fa83a17bfcfe8af69a [term 1 FOLLOWER]: Refusing update from remote peer 5d53cf9c2da74a33bb309bdea1008cd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:42.745334 15617 raft_consensus.cc:1261] T 91c0c0773cc54400a4d4661d74b2ff4d P 9425f4a721a6474bb40adb111db99615 [term 1 FOLLOWER]: Refusing update from remote peer 5d53cf9c2da74a33bb309bdea1008cd5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:42.745477 16044 consensus_queue.cc:1035] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e82ec9077f6244fa83a17bfcfe8af69a" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:42.745620 16046 consensus_queue.cc:1035] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9425f4a721a6474bb40adb111db99615" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35343 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:42.746434 16016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.746317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.1:49817)
0926 15:00:42.746359 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.746393 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 15:00:42.746692 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.746528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.3:57005)
0926 15:00:42.746603 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.746630 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260926 15:00:42.747575 16052 mvcc.cc:205] Tried to move back new op lower bound from 7333621115882123264 to 7333621115736702976. Current Snapshot: MvccSnapshot[applied={T|T < 7333621115882123264}]
I20260926 15:00:42.747710 16045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.747512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.2:58095)
0926 15:00:42.747645 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.747680 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260926 15:00:42.748387 16016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.748281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.1:54701)
0926 15:00:42.748320 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.748355 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260926 15:00:42.749791 16045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.749566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.2:59231)
0926 15:00:42.749730 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.749759 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150}
I20260926 15:00:42.749828 16015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:42.749682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44775 (local address 127.8.102.3:58677)
0926 15:00:42.749769 (+ 87us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.749797 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:44775: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260926 15:00:42.750151 15297 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:42.750245 15297 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49862:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:42.751436 15297 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 26a22fd6ed974376bb5a6d4da9a14957: Sending DeleteTablet for 3 replicas of tablet 91c0c0773cc54400a4d4661d74b2ff4d
I20260926 15:00:42.751742 15597 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c0c0773cc54400a4d4661d74b2ff4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:42 UTC) from {username='oryx', principal='oryx/127.8.102.61@KRBTEST.COM'} at 127.0.0.1:41800
I20260926 15:00:42.751775 15734 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c0c0773cc54400a4d4661d74b2ff4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:42 UTC) from {username='oryx', principal='oryx/127.8.102.61@KRBTEST.COM'} at 127.0.0.1:34656
I20260926 15:00:42.751788 15870 tablet_service.cc:1514] Processing DeleteTablet for tablet 91c0c0773cc54400a4d4661d74b2ff4d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:42 UTC) from {username='oryx', principal='oryx/127.8.102.61@KRBTEST.COM'} at 127.0.0.1:50432
I20260926 15:00:42.751966 16061 tablet_replica.cc:330] T 91c0c0773cc54400a4d4661d74b2ff4d P e82ec9077f6244fa83a17bfcfe8af69a: stopping tablet replica
I20260926 15:00:42.751960 16060 tablet_replica.cc:330] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5: stopping tablet replica
I20260926 15:00:42.752017 16060 raft_consensus.cc:2229] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:00:42.752035 16061 raft_consensus.cc:2229] T 91c0c0773cc54400a4d4661d74b2ff4d P e82ec9077f6244fa83a17bfcfe8af69a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:42.752082 16061 raft_consensus.cc:2258] T 91c0c0773cc54400a4d4661d74b2ff4d P e82ec9077f6244fa83a17bfcfe8af69a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:42.752195 16060 raft_consensus.cc:2258] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:42.752394 16061 ts_tablet_manager.cc:1905] T 91c0c0773cc54400a4d4661d74b2ff4d P e82ec9077f6244fa83a17bfcfe8af69a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:42.752504 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 15444
I20260926 15:00:42.753319 16060 ts_tablet_manager.cc:1905] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:42.754858 16061 ts_tablet_manager.cc:1918] T 91c0c0773cc54400a4d4661d74b2ff4d P e82ec9077f6244fa83a17bfcfe8af69a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:00:42.754920 16061 log.cc:1196] T 91c0c0773cc54400a4d4661d74b2ff4d P e82ec9077f6244fa83a17bfcfe8af69a: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/91c0c0773cc54400a4d4661d74b2ff4d
I20260926 15:00:42.754964 16060 ts_tablet_manager.cc:1918] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:00:42.755028 16060 log.cc:1196] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/91c0c0773cc54400a4d4661d74b2ff4d
I20260926 15:00:42.755165 16061 ts_tablet_manager.cc:1939] T 91c0c0773cc54400a4d4661d74b2ff4d P e82ec9077f6244fa83a17bfcfe8af69a: Deleting consensus metadata
I20260926 15:00:42.755280 16060 ts_tablet_manager.cc:1939] T 91c0c0773cc54400a4d4661d74b2ff4d P 5d53cf9c2da74a33bb309bdea1008cd5: Deleting consensus metadata
I20260926 15:00:42.755637 15282 catalog_manager.cc:4841] TS e82ec9077f6244fa83a17bfcfe8af69a (127.8.102.3:36979): tablet 91c0c0773cc54400a4d4661d74b2ff4d (table test-table [id=22c17930aeb141359f88f564f26f1a94]) successfully deleted
I20260926 15:00:42.755936 15284 catalog_manager.cc:4841] TS 5d53cf9c2da74a33bb309bdea1008cd5 (127.8.102.2:33887): tablet 91c0c0773cc54400a4d4661d74b2ff4d (table test-table [id=22c17930aeb141359f88f564f26f1a94]) successfully deleted
W20260926 15:00:42.761080 16032 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:42.760909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:35343 (local address 127.0.0.1:41812)
0926 15:00:42.760972 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:42.761002 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.1:35343: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 15:00:42.761264 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 15671
W20260926 15:00:42.761291 15281 catalog_manager.cc:4584] TS 9425f4a721a6474bb40adb111db99615 (127.8.102.1:35343): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 91c0c0773cc54400a4d4661d74b2ff4d: Network error: Client connection negotiation failed: client connection to 127.8.102.1:35343: connect: Connection reset by peer (error 104)
I20260926 15:00:42.765930 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 15807
I20260926 15:00:42.770536 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 15164
I20260926 15:00:42.776170 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 15351
2026-09-26T15:00:42Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (6674 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:00:42 dist-test-slave-jnhs krb5kdc[16064](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:00:42 dist-test-slave-jnhs krb5kdc[16064](info): set up 2 sockets
Sep 26 15:00:42 dist-test-slave-jnhs krb5kdc[16064](info): commencing operation
krb5kdc: starting...
W20260926 15:00:43.907243 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.108s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:00:43 dist-test-slave-jnhs krb5kdc[16064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:00:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:00:44Z Disabled control of system clock
WARNING: no policy specified for oryx/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.62.keytab.
Entry for principal oryx/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.62.keytab.
I20260926 15:00:44.035669 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:45815
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.62.keytab
--principal=oryx/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:44509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:45815
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:44.123663 16077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:44.123944 16077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:44.123991 16077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:44.125298 16077 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:44.125377 16077 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260926 15:00:44.125404 16077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:44.125435 16077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:44.125466 16077 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:44.125491 16077 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:00:44.127139 16077 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:44509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:45815
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.62.keytab
--principal=oryx/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:45815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16077
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:44.127431 16077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:44.127667 16077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:44.132802 16085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:44.132795 16082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:44.132795 16083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:44.132814 16077 server_base.cc:1034] running on GCE node
I20260926 15:00:44.133347 16077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:44.133668 16077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:44.134856 16077 hybrid_clock.cc:648] HybridClock initialized: now 1790434844134822 us; error 48 us; skew 500 ppm
Sep 26 15:00:44 dist-test-slave-jnhs krb5kdc[16064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434844, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:44.137570 16077 init.cc:377] Logged in from keytab as oryx/127.8.102.62@KRBTEST.COM (short username oryx)
I20260926 15:00:44.138062 16077 webserver.cc:460] Webserver started at http://127.8.102.62:37579/ using document root <none> and password file <none>
I20260926 15:00:44.138283 16077 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:44.138347 16077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:44.138499 16077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:44.139664 16077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "c5c4bd5ffb5c475ebb429d7e50fe3260"
format_stamp: "Formatted at 2026-09-26 15:00:44 on dist-test-slave-jnhs"
server_key: "4cfe7dc4111134041fe92fa7595abe22"
server_key_iv: "d9b173609f39a56e2c2f5be46f4e27f5"
server_key_version: "encryptionkey@0"
I20260926 15:00:44.140038 16077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "c5c4bd5ffb5c475ebb429d7e50fe3260"
format_stamp: "Formatted at 2026-09-26 15:00:44 on dist-test-slave-jnhs"
server_key: "4cfe7dc4111134041fe92fa7595abe22"
server_key_iv: "d9b173609f39a56e2c2f5be46f4e27f5"
server_key_version: "encryptionkey@0"
I20260926 15:00:44.141353 16077 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 15:00:44.142107 16092 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:44.142274 16077 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 15:00:44.142369 16077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "c5c4bd5ffb5c475ebb429d7e50fe3260"
format_stamp: "Formatted at 2026-09-26 15:00:44 on dist-test-slave-jnhs"
server_key: "4cfe7dc4111134041fe92fa7595abe22"
server_key_iv: "d9b173609f39a56e2c2f5be46f4e27f5"
server_key_version: "encryptionkey@0"
I20260926 15:00:44.142450 16077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:44.153308 16077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:44.155614 16077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:44.155814 16077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:44.160795 16077 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:45815
I20260926 15:00:44.160847 16154 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:45815 every 8 connection(s)
I20260926 15:00:44.161211 16077 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:00:44.161911 16155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:44.164332 16155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260: Bootstrap starting.
I20260926 15:00:44.164950 16155 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:44.165252 16155 log.cc:824] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:44.166066 16155 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260: No bootstrap required, opened a new log
I20260926 15:00:44.168329 16155 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c4bd5ffb5c475ebb429d7e50fe3260" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45815 } }
I20260926 15:00:44.168491 16155 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:44.168514 16155 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5c4bd5ffb5c475ebb429d7e50fe3260, State: Initialized, Role: FOLLOWER
I20260926 15:00:44.168668 16155 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c4bd5ffb5c475ebb429d7e50fe3260" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45815 } }
I20260926 15:00:44.168746 16155 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:00:44.168789 16155 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:00:44.168839 16155 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:44.169466 16155 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c4bd5ffb5c475ebb429d7e50fe3260" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45815 } }
I20260926 15:00:44.169565 16155 leader_election.cc:302] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5c4bd5ffb5c475ebb429d7e50fe3260; no voters:
I20260926 15:00:44.169756 16155 leader_election.cc:288] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:00:44.169857 16158 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:44.170025 16158 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260 [term 1 LEADER]: Becoming Leader. State: Replica: c5c4bd5ffb5c475ebb429d7e50fe3260, State: Running, Role: LEADER
I20260926 15:00:44.170071 16155 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:44.170120 16158 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c4bd5ffb5c475ebb429d7e50fe3260" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45815 } }
I20260926 15:00:44.170578 16159 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5c4bd5ffb5c475ebb429d7e50fe3260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c4bd5ffb5c475ebb429d7e50fe3260" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45815 } } }
I20260926 15:00:44.170696 16159 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:44.170846 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 16077
I20260926 15:00:44.170946 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:00:44.171166 8600 external_mini_cluster.cc:1428] Setting key 66d457ee3b3b1e2e35c3058d73709408
I20260926 15:00:44.171509 16160 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5c4bd5ffb5c475ebb429d7e50fe3260. Latest consensus state: current_term: 1 leader_uuid: "c5c4bd5ffb5c475ebb429d7e50fe3260" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5c4bd5ffb5c475ebb429d7e50fe3260" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45815 } } }
I20260926 15:00:44.171590 16160 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:44.171711 16167 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:00:44.172235 16167 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:00:44.175086 16167 catalog_manager.cc:1347] Generated new cluster ID: 018a74f2bed749b19294dffb97684e57
I20260926 15:00:44.175146 16167 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:00:44.178602 16167 catalog_manager.cc:1370] Generated new certificate authority record
Sep 26 15:00:44 dist-test-slave-jnhs krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434843, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM
I20260926 15:00:44.179349 16167 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:00:44.182984 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:44.171680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60480 (local address 127.8.102.62:45815)
0926 15:00:44.172285 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:44.172293 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:44.172323 (+ 30us) server_negotiation.cc:407] Connection header received
0926 15:00:44.172429 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:44.172435 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:44.172479 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:44.172613 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:44.173395 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.174326 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.175465 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.176118 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.176231 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.179193 (+ 2962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:44.179199 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:44.179199 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:44.179213 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:44.181147 (+ 1934us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:44.181619 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:44.181623 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:44.181624 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:44.181735 (+ 111us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:44.181972 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:44.181973 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:44.181974 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:44.182230 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:44.182450 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:44.182524 (+ 74us) server_negotiation.cc:299] Negotiation successful
0926 15:00:44.182742 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":564,"thread_start_us":74,"threads_started":1}
I20260926 15:00:44.191550 16167 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260: Generated new TSK 0
I20260926 15:00:44.191804 16167 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.1.keytab.
Entry for principal oryx/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.1.keytab.
I20260926 15:00:44.223199 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:45815
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.1.keytab
--principal=oryx/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:44509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:44.311074 16181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:44.311419 16181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:44.311470 16181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:44.312965 16181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:44.313019 16181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:44.313041 16181 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:00:44.313059 16181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:00:44.314716 16181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:44509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.1.keytab
--principal=oryx/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.102.62:45815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:44.314934 16181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:44.315122 16181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:44.320616 16187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:44.320621 16186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:44.320672 16181 server_base.cc:1034] running on GCE node
W20260926 15:00:44.320679 16189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:44.321092 16181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:44.321362 16181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:44.322544 16181 hybrid_clock.cc:648] HybridClock initialized: now 1790434844322522 us; error 37 us; skew 500 ppm
Sep 26 15:00:44 dist-test-slave-jnhs krb5kdc[16064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434844, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:44.325789 16181 init.cc:377] Logged in from keytab as oryx/127.8.102.1@KRBTEST.COM (short username oryx)
I20260926 15:00:44.326334 16181 webserver.cc:460] Webserver started at http://127.8.102.1:34621/ using document root <none> and password file <none>
I20260926 15:00:44.326589 16181 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:44.326686 16181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:44.326840 16181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:44.328048 16181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "1cceaf11e17e46ee9180be9ec63ac941"
format_stamp: "Formatted at 2026-09-26 15:00:44 on dist-test-slave-jnhs"
server_key: "92e291a3f98486e766b594b878bb37ff"
server_key_iv: "d0dfc851cb148ed9e6822b620d298ca7"
server_key_version: "encryptionkey@0"
I20260926 15:00:44.328490 16181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "1cceaf11e17e46ee9180be9ec63ac941"
format_stamp: "Formatted at 2026-09-26 15:00:44 on dist-test-slave-jnhs"
server_key: "92e291a3f98486e766b594b878bb37ff"
server_key_iv: "d0dfc851cb148ed9e6822b620d298ca7"
server_key_version: "encryptionkey@0"
I20260926 15:00:44.330431 16181 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:00:44.331581 16196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:44.331758 16181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:44.331902 16181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "1cceaf11e17e46ee9180be9ec63ac941"
format_stamp: "Formatted at 2026-09-26 15:00:44 on dist-test-slave-jnhs"
server_key: "92e291a3f98486e766b594b878bb37ff"
server_key_iv: "d0dfc851cb148ed9e6822b620d298ca7"
server_key_version: "encryptionkey@0"
I20260926 15:00:44.331992 16181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:44.349087 16181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:44.351297 16181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:44.351418 16181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:44.351878 16181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:44.351948 16181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:44.352005 16181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:44.352032 16181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:44.358601 16181 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:34405
I20260926 15:00:44.358747 16311 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:34405 every 8 connection(s)
I20260926 15:00:44.359059 16181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
Sep 26 15:00:44 dist-test-slave-jnhs krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434844, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM
I20260926 15:00:44.363255 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:44.352428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:57785 (local address 127.8.102.62:45815)
0926 15:00:44.352474 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:44.352477 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:44.352748 (+ 271us) server_negotiation.cc:407] Connection header received
0926 15:00:44.353375 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:44.353376 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:44.353391 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:44.353516 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:44.354538 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.355414 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.357064 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.357431 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.357538 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.360369 (+ 2831us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:44.360371 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:44.360372 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:44.360380 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:44.361698 (+ 1318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:44.362176 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:44.362177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:44.362178 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:44.362245 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:44.362538 (+ 293us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:44.362539 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:44.362540 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:44.362699 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:44.362765 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:44.363054 (+ 289us) server_negotiation.cc:299] Negotiation successful
0926 15:00:44.363162 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 15:00:44.363445 16205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:44.352438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45815 (local address 127.8.102.1:57785)
0926 15:00:44.352621 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:44.352661 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:44.353253 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:44.353610 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:44.353616 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:44.354069 (+ 453us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:44.354399 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:44.354406 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.355523 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.355525 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:44.356961 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:44.356965 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.357550 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.357551 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:44.357724 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.358382 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:44.358394 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:44.360257 (+ 1863us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:44.361818 (+ 1561us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:44.361825 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:44.361828 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:44.362097 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:44.362314 (+ 217us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:44.362315 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:44.362315 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:44.362427 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:44.362763 (+ 336us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:44.362771 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:44.362914 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 15:00:44.363032 (+ 118us) client_negotiation.cc:240] Negotiation successful
0926 15:00:44.363205 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":100,"threads_started":1}
I20260926 15:00:44.364010 16312 heartbeater.cc:347] Connected to a master server at 127.8.102.62:45815
I20260926 15:00:44.364106 16312 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:44.364321 16312 heartbeater.cc:511] Master 127.8.102.62:45815 requested a full tablet report, sending...
I20260926 15:00:44.365021 16109 ts_manager.cc:194] Registered new tserver with Master: 1cceaf11e17e46ee9180be9ec63ac941 (127.8.102.1:34405)
I20260926 15:00:44.365669 16109 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:57785
I20260926 15:00:44.369192 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 16181
I20260926 15:00:44.369333 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:00:44.369627 8600 external_mini_cluster.cc:1428] Setting key b8c8bb89d3aeaccd4c9fbe9252911dd5
I20260926 15:00:44.371582 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:44.365148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:51509 (local address 127.8.102.62:45815)
0926 15:00:44.365217 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:44.365220 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:44.365290 (+ 70us) server_negotiation.cc:407] Connection header received
0926 15:00:44.365370 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:44.365371 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:44.365382 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:44.365524 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:44.365913 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.366434 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.367293 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.367536 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.367598 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.368617 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:44.368618 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:44.368618 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:44.368625 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:44.370205 (+ 1580us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:44.370699 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:44.370701 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:44.370701 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:44.370766 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:44.371036 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:44.371037 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:44.371038 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:44.371205 (+ 167us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:44.371268 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:44.371428 (+ 160us) server_negotiation.cc:299] Negotiation successful
0926 15:00:44.371518 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 15:00:44.371711 16318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:44.365093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45815 (local address 127.8.102.1:51509)
0926 15:00:44.365239 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:44.365249 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:44.365308 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:44.365535 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:44.365536 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:44.365734 (+ 198us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:44.365825 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:44.365828 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.366540 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.366541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:44.367213 (+ 672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:44.367215 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.367627 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.367628 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:44.367689 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.368217 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:44.368223 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:44.368536 (+ 313us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:44.370301 (+ 1765us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:44.370302 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:44.370302 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:44.370580 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:44.370833 (+ 253us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:44.370835 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:44.370835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:44.370914 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:44.371272 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:44.371273 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:44.371330 (+ 57us) client_negotiation.cc:769] Sending connection context
0926 15:00:44.371410 (+ 80us) client_negotiation.cc:240] Negotiation successful
0926 15:00:44.371551 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
WARNING: no policy specified for oryx/127.8.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.2.keytab.
Entry for principal oryx/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.2.keytab.
I20260926 15:00:44.398942 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:45815
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.2.keytab
--principal=oryx/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:44509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:44.484246 16321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:44.484498 16321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:44.484527 16321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:44.485821 16321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:44.485869 16321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:44.485894 16321 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:00:44.485914 16321 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:00:44.487649 16321 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:44509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.2.keytab
--principal=oryx/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.102.62:45815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16321
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:44.487890 16321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:44.488099 16321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:44.493431 16327 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:44.493350 16326 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:44.493463 16321 server_base.cc:1034] running on GCE node
W20260926 15:00:44.493606 16329 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:44.493830 16321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:44.494105 16321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:44.495277 16321 hybrid_clock.cc:648] HybridClock initialized: now 1790434844495255 us; error 38 us; skew 500 ppm
Sep 26 15:00:44 dist-test-slave-jnhs krb5kdc[16064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434844, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:44.498085 16321 init.cc:377] Logged in from keytab as oryx/127.8.102.2@KRBTEST.COM (short username oryx)
I20260926 15:00:44.498566 16321 webserver.cc:460] Webserver started at http://127.8.102.2:36089/ using document root <none> and password file <none>
I20260926 15:00:44.498910 16321 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:44.499027 16321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:44.499203 16321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:44.500380 16321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "7aad28146b09442eace3bcf19cc525ea"
format_stamp: "Formatted at 2026-09-26 15:00:44 on dist-test-slave-jnhs"
server_key: "304079f5adf6456c986bc9e512cba29d"
server_key_iv: "6a8680b750d01921ee789733ca07a99a"
server_key_version: "encryptionkey@0"
I20260926 15:00:44.500756 16321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "7aad28146b09442eace3bcf19cc525ea"
format_stamp: "Formatted at 2026-09-26 15:00:44 on dist-test-slave-jnhs"
server_key: "304079f5adf6456c986bc9e512cba29d"
server_key_iv: "6a8680b750d01921ee789733ca07a99a"
server_key_version: "encryptionkey@0"
I20260926 15:00:44.502279 16321 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:00:44.503163 16336 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:44.503381 16321 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 15:00:44.503497 16321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "7aad28146b09442eace3bcf19cc525ea"
format_stamp: "Formatted at 2026-09-26 15:00:44 on dist-test-slave-jnhs"
server_key: "304079f5adf6456c986bc9e512cba29d"
server_key_iv: "6a8680b750d01921ee789733ca07a99a"
server_key_version: "encryptionkey@0"
I20260926 15:00:44.503578 16321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:44.516466 16321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:44.518962 16321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:44.519080 16321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:44.519560 16321 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:44.519599 16321 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:44.519626 16321 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:44.519672 16321 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:00:44 dist-test-slave-jnhs krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434844, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM
I20260926 15:00:44.526796 16321 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:45043
I20260926 15:00:44.526880 16451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:45043 every 8 connection(s)
I20260926 15:00:44.527216 16321 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:00:44.529963 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:44.520109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:46889 (local address 127.8.102.62:45815)
0926 15:00:44.520157 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:44.520159 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:44.520392 (+ 233us) server_negotiation.cc:407] Connection header received
0926 15:00:44.520945 (+ 553us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:44.520947 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:44.520960 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:44.521028 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:44.522031 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.522826 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.523848 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.524068 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.524164 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.527112 (+ 2948us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:44.527114 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:44.527114 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:44.527121 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:44.528616 (+ 1495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:44.529018 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:44.529018 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:44.529019 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:44.529071 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:44.529372 (+ 301us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:44.529373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:44.529373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:44.529519 (+ 146us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:44.529573 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:44.529831 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 15:00:44.529898 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 15:00:44.530155 16345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:44.520114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45815 (local address 127.8.102.2:46889)
0926 15:00:44.520267 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:44.520305 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:44.520827 (+ 522us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:44.521143 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:44.521149 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:44.521575 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:44.521902 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:44.521910 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.522931 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.522932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:44.523753 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:44.523755 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.524218 (+ 463us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.524220 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:44.524412 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.525026 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:44.525046 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:44.526993 (+ 1947us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:44.528705 (+ 1712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:44.528712 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:44.528714 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:44.528946 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:44.529154 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:44.529155 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:44.529156 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:44.529308 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:44.529584 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:44.529589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:44.529712 (+ 123us) client_negotiation.cc:769] Sending connection context
0926 15:00:44.529808 (+ 96us) client_negotiation.cc:240] Negotiation successful
0926 15:00:44.529933 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":87,"threads_started":1}
I20260926 15:00:44.530750 16452 heartbeater.cc:347] Connected to a master server at 127.8.102.62:45815
I20260926 15:00:44.530838 16452 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:44.531035 16452 heartbeater.cc:511] Master 127.8.102.62:45815 requested a full tablet report, sending...
I20260926 15:00:44.531436 16109 ts_manager.cc:194] Registered new tserver with Master: 7aad28146b09442eace3bcf19cc525ea (127.8.102.2:45043)
I20260926 15:00:44.531924 16109 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:46889
I20260926 15:00:44.534435 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 16321
I20260926 15:00:44.534551 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:00:44.534797 8600 external_mini_cluster.cc:1428] Setting key 1a6a53df87dc6f46b241e3cf38e188b7
I20260926 15:00:44.537859 16458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:44.531761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45815 (local address 127.8.102.2:33687)
0926 15:00:44.531918 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:44.531927 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:44.531994 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:44.532168 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:44.532169 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:44.532353 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:44.532407 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:44.532409 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.533040 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.533041 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:44.533549 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:44.533551 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.533905 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.533906 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:44.533984 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.534398 (+ 414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:44.534403 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:44.534851 (+ 448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:44.536637 (+ 1786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:44.536638 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:44.536638 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:44.536895 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:44.537131 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:44.537131 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:44.537132 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:44.537191 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:44.537511 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:44.537512 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:44.537567 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 15:00:44.537650 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 15:00:44.537772 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260926 15:00:44.537902 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:44.531814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:33687 (local address 127.8.102.62:45815)
0926 15:00:44.531856 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:44.531857 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:44.531980 (+ 123us) server_negotiation.cc:407] Connection header received
0926 15:00:44.532093 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:44.532093 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:44.532103 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:44.532163 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:44.532488 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.532954 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.533628 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.533833 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.533908 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.534976 (+ 1068us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:44.534977 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:44.534977 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:44.534983 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:44.536565 (+ 1582us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:44.536997 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:44.536998 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:44.536999 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:44.537058 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:44.537257 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:44.537258 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:44.537259 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:44.537408 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:44.537487 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:44.537685 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 15:00:44.537768 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
WARNING: no policy specified for oryx/127.8.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.3.keytab.
Entry for principal oryx/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.3.keytab.
I20260926 15:00:44.563577 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:45815
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.3.keytab
--principal=oryx/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:44509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:44.646845 16461 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:44.647061 16461 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:44.647087 16461 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:44.648327 16461 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:44.648372 16461 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:44.648393 16461 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260926 15:00:44.648419 16461 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:00:44.649963 16461 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:44509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.3.keytab
--principal=oryx/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.8.102.62:45815
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16461
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:44.650164 16461 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:44.650363 16461 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:44.656056 16469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:44.656044 16467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:44.656050 16466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:44.656140 16461 server_base.cc:1034] running on GCE node
I20260926 15:00:44.656584 16461 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:44.656862 16461 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:44.658041 16461 hybrid_clock.cc:648] HybridClock initialized: now 1790434844658020 us; error 41 us; skew 500 ppm
Sep 26 15:00:44 dist-test-slave-jnhs krb5kdc[16064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434844, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:44.660617 16461 init.cc:377] Logged in from keytab as oryx/127.8.102.3@KRBTEST.COM (short username oryx)
I20260926 15:00:44.661072 16461 webserver.cc:460] Webserver started at http://127.8.102.3:37235/ using document root <none> and password file <none>
I20260926 15:00:44.661303 16461 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:44.661369 16461 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:44.661521 16461 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:44.662580 16461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "efb7c3b9725645e786dc0797eb8fbfe6"
format_stamp: "Formatted at 2026-09-26 15:00:44 on dist-test-slave-jnhs"
server_key: "63a2676f3fcb75e622559ad6e5b4d134"
server_key_iv: "bce57354a4b9af4efad4b36450d3d96e"
server_key_version: "encryptionkey@0"
I20260926 15:00:44.663003 16461 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "efb7c3b9725645e786dc0797eb8fbfe6"
format_stamp: "Formatted at 2026-09-26 15:00:44 on dist-test-slave-jnhs"
server_key: "63a2676f3fcb75e622559ad6e5b4d134"
server_key_iv: "bce57354a4b9af4efad4b36450d3d96e"
server_key_version: "encryptionkey@0"
I20260926 15:00:44.664542 16461 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:44.665469 16476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:44.665678 16461 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:44.665789 16461 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "efb7c3b9725645e786dc0797eb8fbfe6"
format_stamp: "Formatted at 2026-09-26 15:00:44 on dist-test-slave-jnhs"
server_key: "63a2676f3fcb75e622559ad6e5b4d134"
server_key_iv: "bce57354a4b9af4efad4b36450d3d96e"
server_key_version: "encryptionkey@0"
I20260926 15:00:44.665884 16461 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:44.692183 16461 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:44.694825 16461 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:44.695030 16461 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:44.695670 16461 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:44.695737 16461 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:44.695775 16461 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:44.695813 16461 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 26 15:00:44 dist-test-slave-jnhs krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434844, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM
I20260926 15:00:44.703547 16461 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:44483
I20260926 15:00:44.703624 16591 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:44483 every 8 connection(s)
I20260926 15:00:44.704010 16461 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:00:44.707047 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:44.696143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:34701 (local address 127.8.102.62:45815)
0926 15:00:44.696187 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:44.696193 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:44.696442 (+ 249us) server_negotiation.cc:407] Connection header received
0926 15:00:44.697000 (+ 558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:44.697002 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:44.697015 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:44.697100 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:44.698003 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.699006 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.700103 (+ 1097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.700421 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.700500 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.703722 (+ 3222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:44.703724 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:44.703725 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:44.703734 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:44.705391 (+ 1657us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:44.705844 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:44.705845 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:44.705846 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:44.705912 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:44.706231 (+ 319us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:44.706232 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:44.706233 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:44.706353 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:44.706408 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:44.706899 (+ 491us) server_negotiation.cc:299] Negotiation successful
0926 15:00:44.706980 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 15:00:44.707209 16485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:44.696160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45815 (local address 127.8.102.3:34701)
0926 15:00:44.696300 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:44.696344 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:44.696896 (+ 552us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:44.697180 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:44.697183 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:44.697575 (+ 392us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:44.697873 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:44.697881 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.699125 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.699127 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:44.699989 (+ 862us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:44.699991 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.700538 (+ 547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.700539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:44.700731 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.701398 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:44.701410 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:44.703538 (+ 2128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:44.705491 (+ 1953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:44.705498 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:44.705500 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:44.705744 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:44.706019 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:44.706020 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:44.706021 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:44.706162 (+ 141us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:44.706414 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:44.706419 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:44.706691 (+ 272us) client_negotiation.cc:769] Sending connection context
0926 15:00:44.706826 (+ 135us) client_negotiation.cc:240] Negotiation successful
0926 15:00:44.706998 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":83,"threads_started":1}
I20260926 15:00:44.707746 16592 heartbeater.cc:347] Connected to a master server at 127.8.102.62:45815
I20260926 15:00:44.707859 16592 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:44.708053 16592 heartbeater.cc:511] Master 127.8.102.62:45815 requested a full tablet report, sending...
I20260926 15:00:44.708511 16108 ts_manager.cc:194] Registered new tserver with Master: efb7c3b9725645e786dc0797eb8fbfe6 (127.8.102.3:44483)
I20260926 15:00:44.709123 16108 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:34701
I20260926 15:00:44.709160 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 16461
I20260926 15:00:44.709256 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:00:44.709478 8600 external_mini_cluster.cc:1428] Setting key 49884d4515e15fcc087fb0fccf9efb1e
I20260926 15:00:44.710386 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:00:44.715804 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:44.708931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:39287 (local address 127.8.102.62:45815)
0926 15:00:44.708985 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:44.708987 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:44.709000 (+ 13us) server_negotiation.cc:407] Connection header received
0926 15:00:44.709068 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:44.709069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:44.709080 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:44.709163 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:44.709663 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.710398 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.711245 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.711560 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.711638 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.712774 (+ 1136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:44.712776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:44.712776 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:44.712786 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:44.714500 (+ 1714us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:44.715001 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:44.715002 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:44.715002 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:44.715060 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:44.715267 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:44.715268 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:44.715268 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:44.715376 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:44.715427 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:44.715608 (+ 181us) server_negotiation.cc:299] Negotiation successful
0926 15:00:44.715674 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 15:00:44.715891 16598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:44.708803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45815 (local address 127.8.102.3:39287)
0926 15:00:44.708954 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:44.708964 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:44.709013 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:44.709170 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:44.709171 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:44.709486 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:44.709583 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:44.709587 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.710510 (+ 923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.710511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:44.711161 (+ 650us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:44.711163 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.711644 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.711644 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:44.711698 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.712294 (+ 596us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:44.712301 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:44.712671 (+ 370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:44.714601 (+ 1930us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:44.714602 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:44.714603 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:44.714901 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:44.715134 (+ 233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:44.715136 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:44.715136 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:44.715212 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:44.715439 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:44.715440 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:44.715506 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 15:00:44.715591 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 15:00:44.715715 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
Sep 26 15:00:44 dist-test-slave-jnhs krb5kdc[16064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434844, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 26 15:00:44 dist-test-slave-jnhs krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:44 dist-test-slave-jnhs krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 15:00:44.739974 16174 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:44.732349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60488 (local address 127.8.102.62:45815)
0926 15:00:44.732400 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:44.732403 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:44.732562 (+ 159us) server_negotiation.cc:407] Connection header received
0926 15:00:44.732678 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:44.732679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:44.732692 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:44.732725 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:44.733611 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.734456 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.735669 (+ 1213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.735960 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.736034 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.739848 (+ 3814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60488: BlockingRecv error: recv got EOF from 127.0.0.1:60488 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 26 15:00:44 dist-test-slave-jnhs krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434844, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM
I20260926 15:00:44.747705 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:44.740810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60504 (local address 127.8.102.62:45815)
0926 15:00:44.740886 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:44.740889 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:44.740935 (+ 46us) server_negotiation.cc:407] Connection header received
0926 15:00:44.741014 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:44.741015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:44.741025 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:44.741075 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:44.741555 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.742043 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.742842 (+ 799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.743040 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.743118 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.745256 (+ 2138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:44.745258 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:44.745259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:44.745266 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:44.746522 (+ 1256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:44.746885 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:44.746887 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:44.746887 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:44.746962 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:44.747153 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:44.747154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:44.747155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:44.747267 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:44.747372 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:44.747516 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 15:00:44.747600 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:00:44.749588 16108 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:00:44.751490 16108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:00:44.758132 16614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:44.753866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:45043 (local address 127.0.0.1:54986)
0926 15:00:44.754149 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:44.754195 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:44.754278 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:44.754593 (+ 315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:44.754596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:44.754599 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:44.754704 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:44.754706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.756035 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.756035 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:44.756958 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:44.756961 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.757802 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.757803 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:44.757892 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.757909 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 15:00:44.757984 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 15:00:44.758038 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":245,"mutex_wait_us":34,"thread_start_us":60,"threads_started":1}
I20260926 15:00:44.758157 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:44.753948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54986 (local address 127.8.102.2:45043)
0926 15:00:44.754143 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:44.754148 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:44.754267 (+ 119us) server_negotiation.cc:407] Connection header received
0926 15:00:44.754475 (+ 208us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:44.754479 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:44.754496 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:44.754573 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:44.754802 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.755917 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.757058 (+ 1141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.757719 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.757796 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.757871 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:44.758009 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 15:00:44.758057 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":90,"threads_started":1}
I20260926 15:00:44.758607 16615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:44.753897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:44483 (local address 127.0.0.1:53326)
0926 15:00:44.754387 (+ 490us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:44.754396 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:44.754454 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:44.754682 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:44.754683 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:44.754685 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:44.754743 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:44.754746 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.756001 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.756002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:44.757331 (+ 1329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:44.757333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.758369 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.758370 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:44.758431 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.758434 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:00:44.758482 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 15:00:44.758532 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"mutex_wait_us":70,"thread_start_us":88,"threads_started":1}
I20260926 15:00:44.758809 16617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:44.754079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:34405 (local address 127.0.0.1:38408)
0926 15:00:44.754866 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:44.754875 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:44.754918 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:44.755177 (+ 259us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:44.755178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:44.755180 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:44.755231 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:44.755233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.756103 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.756104 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:44.757565 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:44.757567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.758537 (+ 970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.758538 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:44.758598 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.758603 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:00:44.758690 (+ 87us) client_negotiation.cc:240] Negotiation successful
0926 15:00:44.758734 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":770,"mutex_wait_us":72,"thread_start_us":64,"threads_started":1}
I20260926 15:00:44.758915 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:44.754207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53326 (local address 127.8.102.3:44483)
0926 15:00:44.754420 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:44.754424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:44.754435 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:00:44.754526 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:44.754528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:44.754541 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:44.754692 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:44.754815 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.755910 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.757420 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.758184 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.758258 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.758335 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:44.758730 (+ 395us) server_negotiation.cc:299] Negotiation successful
0926 15:00:44.758777 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":90,"threads_started":1}
I20260926 15:00:44.758852 16619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:44.754203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38408 (local address 127.8.102.1:34405)
0926 15:00:44.754416 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:44.754421 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:44.754982 (+ 561us) server_negotiation.cc:407] Connection header received
0926 15:00:44.755077 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:44.755079 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:44.755091 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:44.755169 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:44.755285 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.756012 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.757641 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:44.758456 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:44.758536 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:44.758603 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:44.758706 (+ 103us) server_negotiation.cc:299] Negotiation successful
0926 15:00:44.758744 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":185,"thread_start_us":108,"threads_started":1}
I20260926 15:00:44.759624 16526 tablet_service.cc:1467] Processing CreateTablet for tablet ef23f078457d411d93b12b4e5fbbf966 (DEFAULT_TABLE table=test-table [id=078a2ef3026f4d6e91d5820c34250cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:44.759721 16246 tablet_service.cc:1467] Processing CreateTablet for tablet ef23f078457d411d93b12b4e5fbbf966 (DEFAULT_TABLE table=test-table [id=078a2ef3026f4d6e91d5820c34250cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:44.759732 16386 tablet_service.cc:1467] Processing CreateTablet for tablet ef23f078457d411d93b12b4e5fbbf966 (DEFAULT_TABLE table=test-table [id=078a2ef3026f4d6e91d5820c34250cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:44.760025 16526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef23f078457d411d93b12b4e5fbbf966. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:44.760054 16386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef23f078457d411d93b12b4e5fbbf966. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:44.760054 16246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ef23f078457d411d93b12b4e5fbbf966. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:44.763015 16620 tablet_bootstrap.cc:492] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6: Bootstrap starting.
I20260926 15:00:44.763765 16620 tablet_bootstrap.cc:654] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:44.763870 16621 tablet_bootstrap.cc:492] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941: Bootstrap starting.
I20260926 15:00:44.764333 16620 log.cc:824] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:44.764528 16621 tablet_bootstrap.cc:654] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:44.764810 16622 tablet_bootstrap.cc:492] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea: Bootstrap starting.
I20260926 15:00:44.764911 16621 log.cc:824] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:44.765369 16620 tablet_bootstrap.cc:492] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6: No bootstrap required, opened a new log
I20260926 15:00:44.765455 16620 ts_tablet_manager.cc:1397] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 15:00:44.765633 16621 tablet_bootstrap.cc:492] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941: No bootstrap required, opened a new log
I20260926 15:00:44.765697 16621 ts_tablet_manager.cc:1397] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:00:44.767078 16622 tablet_bootstrap.cc:654] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:44.767397 16622 log.cc:824] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:44.767546 16620 raft_consensus.cc:348] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:44.767737 16620 raft_consensus.cc:374] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:44.767788 16620 raft_consensus.cc:729] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb7c3b9725645e786dc0797eb8fbfe6, State: Initialized, Role: FOLLOWER
I20260926 15:00:44.767745 16621 raft_consensus.cc:348] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:44.767906 16621 raft_consensus.cc:374] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:44.767946 16621 raft_consensus.cc:729] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cceaf11e17e46ee9180be9ec63ac941, State: Initialized, Role: FOLLOWER
I20260926 15:00:44.767944 16620 consensus_queue.cc:260] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:44.768056 16621 consensus_queue.cc:260] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:44.768234 16622 tablet_bootstrap.cc:492] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea: No bootstrap required, opened a new log
I20260926 15:00:44.768244 16620 ts_tablet_manager.cc:1428] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:00:44.768297 16592 heartbeater.cc:503] Master 127.8.102.62:45815 was elected leader, sending a full tablet report...
I20260926 15:00:44.768318 16622 ts_tablet_manager.cc:1397] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260926 15:00:44.768457 16312 heartbeater.cc:503] Master 127.8.102.62:45815 was elected leader, sending a full tablet report...
I20260926 15:00:44.768640 16621 ts_tablet_manager.cc:1428] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:00:44.770630 16622 raft_consensus.cc:348] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:44.770851 16622 raft_consensus.cc:374] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:44.770901 16622 raft_consensus.cc:729] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aad28146b09442eace3bcf19cc525ea, State: Initialized, Role: FOLLOWER
I20260926 15:00:44.771054 16622 consensus_queue.cc:260] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:44.771359 16622 ts_tablet_manager.cc:1428] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 15:00:44.771390 16452 heartbeater.cc:503] Master 127.8.102.62:45815 was elected leader, sending a full tablet report...
W20260926 15:00:44.777940 16453 tablet.cc:2369] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:00:44.859958 16313 tablet.cc:2369] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:00:44.954704 16593 tablet.cc:2369] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:00:45.078135 16628 raft_consensus.cc:482] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:45.078325 16628 raft_consensus.cc:504] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.078790 16628 leader_election.cc:288] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cceaf11e17e46ee9180be9ec63ac941 (127.8.102.1:34405), efb7c3b9725645e786dc0797eb8fbfe6 (127.8.102.3:44483)
I20260926 15:00:45.081817 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.079128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:53567 (local address 127.8.102.3:44483)
0926 15:00:45.079192 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:45.079195 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:45.079224 (+ 29us) server_negotiation.cc:407] Connection header received
0926 15:00:45.079318 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:45.079320 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:45.079331 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:45.079400 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:45.079575 (+ 175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.080037 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.081040 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.081459 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.081541 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.081551 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:45.081706 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 15:00:45.081742 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 15:00:45.081836 16629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.078990 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:44483 (local address 127.8.102.2:53567)
0926 15:00:45.079156 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:45.079167 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:45.079243 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:45.079402 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:45.079403 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:45.079405 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:45.079499 (+ 94us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:45.079501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.080127 (+ 626us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.080128 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:45.080945 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:45.080947 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.081542 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.081543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:45.081625 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.081628 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:00:45.081698 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 15:00:45.081742 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"mutex_wait_us":27,"thread_start_us":61,"threads_started":1}
I20260926 15:00:45.081990 16619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.079045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:60671 (local address 127.8.102.1:34405)
0926 15:00:45.079095 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:45.079097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:45.079236 (+ 139us) server_negotiation.cc:407] Connection header received
0926 15:00:45.079310 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:45.079311 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:45.079330 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:45.079401 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:45.079565 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.080091 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.081109 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.081619 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.081701 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.081712 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:45.081878 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 15:00:45.081916 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 15:00:45.082000 16630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.078986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:34405 (local address 127.8.102.2:60671)
0926 15:00:45.079143 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:45.079157 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:45.079245 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:45.079403 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:45.079404 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:45.079406 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:45.079489 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:45.079491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.080181 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.080182 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:45.080988 (+ 806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:45.080990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.081717 (+ 727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.081718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:45.081790 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.081794 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:00:45.081869 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 15:00:45.081910 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"mutex_wait_us":3,"thread_start_us":92,"threads_started":1}
I20260926 15:00:45.082325 16546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef23f078457d411d93b12b4e5fbbf966" candidate_uuid: "7aad28146b09442eace3bcf19cc525ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" is_pre_election: true
I20260926 15:00:45.082353 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef23f078457d411d93b12b4e5fbbf966" candidate_uuid: "7aad28146b09442eace3bcf19cc525ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cceaf11e17e46ee9180be9ec63ac941" is_pre_election: true
I20260926 15:00:45.082497 16546 raft_consensus.cc:2454] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7aad28146b09442eace3bcf19cc525ea in term 0.
I20260926 15:00:45.082497 16266 raft_consensus.cc:2454] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7aad28146b09442eace3bcf19cc525ea in term 0.
I20260926 15:00:45.082748 16338 leader_election.cc:302] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7aad28146b09442eace3bcf19cc525ea, efb7c3b9725645e786dc0797eb8fbfe6; no voters:
I20260926 15:00:45.082970 16628 raft_consensus.cc:2781] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:00:45.083055 16628 raft_consensus.cc:482] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:00:45.083093 16628 raft_consensus.cc:3037] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:45.083851 16628 raft_consensus.cc:504] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.084008 16628 leader_election.cc:288] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [CANDIDATE]: Term 1 election: Requested vote from peers 1cceaf11e17e46ee9180be9ec63ac941 (127.8.102.1:34405), efb7c3b9725645e786dc0797eb8fbfe6 (127.8.102.3:44483)
I20260926 15:00:45.084168 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef23f078457d411d93b12b4e5fbbf966" candidate_uuid: "7aad28146b09442eace3bcf19cc525ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cceaf11e17e46ee9180be9ec63ac941"
I20260926 15:00:45.084190 16546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ef23f078457d411d93b12b4e5fbbf966" candidate_uuid: "7aad28146b09442eace3bcf19cc525ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb7c3b9725645e786dc0797eb8fbfe6"
I20260926 15:00:45.084251 16266 raft_consensus.cc:3037] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:45.084280 16546 raft_consensus.cc:3037] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:45.084960 16546 raft_consensus.cc:2454] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7aad28146b09442eace3bcf19cc525ea in term 1.
I20260926 15:00:45.084985 16266 raft_consensus.cc:2454] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7aad28146b09442eace3bcf19cc525ea in term 1.
I20260926 15:00:45.085162 16340 leader_election.cc:302] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cceaf11e17e46ee9180be9ec63ac941, 7aad28146b09442eace3bcf19cc525ea; no voters:
I20260926 15:00:45.085260 16628 raft_consensus.cc:2781] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:45.085372 16628 raft_consensus.cc:686] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [term 1 LEADER]: Becoming Leader. State: Replica: 7aad28146b09442eace3bcf19cc525ea, State: Running, Role: LEADER
I20260926 15:00:45.085458 16628 consensus_queue.cc:237] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.086316 16109 catalog_manager.cc:5495] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 7aad28146b09442eace3bcf19cc525ea (127.8.102.2). New cstate: current_term: 1 leader_uuid: "7aad28146b09442eace3bcf19cc525ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } health_report { overall_health: UNKNOWN } } }
I20260926 15:00:45.113765 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.111087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54992 (local address 127.8.102.2:45043)
0926 15:00:45.111154 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:45.111156 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:45.111168 (+ 12us) server_negotiation.cc:407] Connection header received
0926 15:00:45.111219 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:45.111220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:45.111242 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:45.111331 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:45.111535 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.112057 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.112892 (+ 835us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.113109 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.113189 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.113411 (+ 222us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:45.113519 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:45.113620 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:45.113641 (+ 21us) server_negotiation.cc:299] Negotiation successful
0926 15:00:45.113664 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 15:00:45.115836 16266 raft_consensus.cc:1261] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941 [term 1 FOLLOWER]: Refusing update from remote peer 7aad28146b09442eace3bcf19cc525ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:45.115859 16546 raft_consensus.cc:1261] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6 [term 1 FOLLOWER]: Refusing update from remote peer 7aad28146b09442eace3bcf19cc525ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:45.116222 16631 consensus_queue.cc:1035] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:45.116369 16628 consensus_queue.cc:1035] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:45.119138 16636 mvcc.cc:205] Tried to move back new op lower bound from 7333621125591699456 to 7333621125470838784. Current Snapshot: MvccSnapshot[applied={T|T < 7333621125591699456}]
I20260926 15:00:45.119212 16635 mvcc.cc:205] Tried to move back new op lower bound from 7333621125591699456 to 7333621125470838784. Current Snapshot: MvccSnapshot[applied={T|T < 7333621125591699456 or (T in {7333621125591699456})}]
I20260926 15:00:45.121768 16109 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:45.121879 16109 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:45.122690 16109 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260: Sending DeleteTablet for 3 replicas of tablet ef23f078457d411d93b12b4e5fbbf966
I20260926 15:00:45.123061 16526 tablet_service.cc:1514] Processing DeleteTablet for tablet ef23f078457d411d93b12b4e5fbbf966 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:45 UTC) from {username='oryx', principal='oryx/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:53326
I20260926 15:00:45.123109 16386 tablet_service.cc:1514] Processing DeleteTablet for tablet ef23f078457d411d93b12b4e5fbbf966 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:45 UTC) from {username='oryx', principal='oryx/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:54986
I20260926 15:00:45.123323 16246 tablet_service.cc:1514] Processing DeleteTablet for tablet ef23f078457d411d93b12b4e5fbbf966 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:45 UTC) from {username='oryx', principal='oryx/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:38408
I20260926 15:00:45.123528 16646 tablet_replica.cc:330] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941: stopping tablet replica
I20260926 15:00:45.123551 16645 tablet_replica.cc:330] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6: stopping tablet replica
I20260926 15:00:45.123656 16645 raft_consensus.cc:2229] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:45.123692 16646 raft_consensus.cc:2229] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:45.123840 16646 raft_consensus.cc:2258] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:45.123760 16109 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60504:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:00:45.124001 16109 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:00:45.123978 16645 raft_consensus.cc:2258] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:45.124396 16646 ts_tablet_manager.cc:1905] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:45.124626 16645 ts_tablet_manager.cc:1905] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:45.125888 16526 tablet_service.cc:1467] Processing CreateTablet for tablet 6926963e08804af89f5997aa409e2496 (DEFAULT_TABLE table=test-table [id=2d7d5eda104d46c8bbd3b2ae341d49fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:45.125998 16526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6926963e08804af89f5997aa409e2496. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:45.126662 16646 ts_tablet_manager.cc:1918] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:00:45.126775 16646 log.cc:1196] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-0/wal/wals/ef23f078457d411d93b12b4e5fbbf966
I20260926 15:00:45.126989 16645 ts_tablet_manager.cc:1918] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:00:45.127079 16645 log.cc:1196] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/ef23f078457d411d93b12b4e5fbbf966
I20260926 15:00:45.127163 16646 ts_tablet_manager.cc:1939] T ef23f078457d411d93b12b4e5fbbf966 P 1cceaf11e17e46ee9180be9ec63ac941: Deleting consensus metadata
I20260926 15:00:45.127261 16386 tablet_service.cc:1467] Processing CreateTablet for tablet 6926963e08804af89f5997aa409e2496 (DEFAULT_TABLE table=test-table [id=2d7d5eda104d46c8bbd3b2ae341d49fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:45.127381 16645 ts_tablet_manager.cc:1939] T ef23f078457d411d93b12b4e5fbbf966 P efb7c3b9725645e786dc0797eb8fbfe6: Deleting consensus metadata
I20260926 15:00:45.127393 16386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6926963e08804af89f5997aa409e2496. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:45.127764 16096 catalog_manager.cc:4841] TS 1cceaf11e17e46ee9180be9ec63ac941 (127.8.102.1:34405): tablet ef23f078457d411d93b12b4e5fbbf966 (table test-table [id=078a2ef3026f4d6e91d5820c34250cc6]) successfully deleted
I20260926 15:00:45.127841 16644 tablet_replica.cc:330] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea: stopping tablet replica
I20260926 15:00:45.128003 16644 raft_consensus.cc:2229] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:00:45.128165 16644 raft_consensus.cc:2258] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:45.128736 16622 tablet_bootstrap.cc:492] T 6926963e08804af89f5997aa409e2496 P 7aad28146b09442eace3bcf19cc525ea: Bootstrap starting.
I20260926 15:00:45.128921 16094 catalog_manager.cc:4841] TS efb7c3b9725645e786dc0797eb8fbfe6 (127.8.102.3:44483): tablet ef23f078457d411d93b12b4e5fbbf966 (table test-table [id=078a2ef3026f4d6e91d5820c34250cc6]) successfully deleted
I20260926 15:00:45.129142 16246 tablet_service.cc:1467] Processing CreateTablet for tablet 6926963e08804af89f5997aa409e2496 (DEFAULT_TABLE table=test-table [id=2d7d5eda104d46c8bbd3b2ae341d49fe]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:45.129225 16622 tablet_bootstrap.cc:654] T 6926963e08804af89f5997aa409e2496 P 7aad28146b09442eace3bcf19cc525ea: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:45.129263 16246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6926963e08804af89f5997aa409e2496. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:45.129942 16622 tablet_bootstrap.cc:492] T 6926963e08804af89f5997aa409e2496 P 7aad28146b09442eace3bcf19cc525ea: No bootstrap required, opened a new log
I20260926 15:00:45.130002 16622 ts_tablet_manager.cc:1397] T 6926963e08804af89f5997aa409e2496 P 7aad28146b09442eace3bcf19cc525ea: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:45.130173 16622 raft_consensus.cc:348] T 6926963e08804af89f5997aa409e2496 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.130259 16622 raft_consensus.cc:374] T 6926963e08804af89f5997aa409e2496 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:45.130281 16622 raft_consensus.cc:729] T 6926963e08804af89f5997aa409e2496 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aad28146b09442eace3bcf19cc525ea, State: Initialized, Role: FOLLOWER
I20260926 15:00:45.130335 16622 consensus_queue.cc:260] T 6926963e08804af89f5997aa409e2496 P 7aad28146b09442eace3bcf19cc525ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.130425 16622 ts_tablet_manager.cc:1428] T 6926963e08804af89f5997aa409e2496 P 7aad28146b09442eace3bcf19cc525ea: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:45.130782 16621 tablet_bootstrap.cc:492] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941: Bootstrap starting.
I20260926 15:00:45.131174 16621 tablet_bootstrap.cc:654] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:45.131367 16620 tablet_bootstrap.cc:492] T 6926963e08804af89f5997aa409e2496 P efb7c3b9725645e786dc0797eb8fbfe6: Bootstrap starting.
I20260926 15:00:45.131824 16620 tablet_bootstrap.cc:654] T 6926963e08804af89f5997aa409e2496 P efb7c3b9725645e786dc0797eb8fbfe6: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:45.132252 16644 ts_tablet_manager.cc:1905] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:45.132616 16621 tablet_bootstrap.cc:492] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941: No bootstrap required, opened a new log
I20260926 15:00:45.132664 16621 ts_tablet_manager.cc:1397] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:00:45.132732 16620 tablet_bootstrap.cc:492] T 6926963e08804af89f5997aa409e2496 P efb7c3b9725645e786dc0797eb8fbfe6: No bootstrap required, opened a new log
I20260926 15:00:45.132777 16620 ts_tablet_manager.cc:1397] T 6926963e08804af89f5997aa409e2496 P efb7c3b9725645e786dc0797eb8fbfe6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:45.132802 16621 raft_consensus.cc:348] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.132863 16621 raft_consensus.cc:374] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:45.132884 16621 raft_consensus.cc:729] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cceaf11e17e46ee9180be9ec63ac941, State: Initialized, Role: FOLLOWER
I20260926 15:00:45.132920 16621 consensus_queue.cc:260] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.132949 16620 raft_consensus.cc:348] T 6926963e08804af89f5997aa409e2496 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.133020 16620 raft_consensus.cc:374] T 6926963e08804af89f5997aa409e2496 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:45.133025 16621 ts_tablet_manager.cc:1428] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:45.133042 16620 raft_consensus.cc:729] T 6926963e08804af89f5997aa409e2496 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb7c3b9725645e786dc0797eb8fbfe6, State: Initialized, Role: FOLLOWER
I20260926 15:00:45.133085 16620 consensus_queue.cc:260] T 6926963e08804af89f5997aa409e2496 P efb7c3b9725645e786dc0797eb8fbfe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.133174 16620 ts_tablet_manager.cc:1428] T 6926963e08804af89f5997aa409e2496 P efb7c3b9725645e786dc0797eb8fbfe6: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 15:00:45.134783 16644 ts_tablet_manager.cc:1918] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:00:45.134857 16644 log.cc:1196] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/ef23f078457d411d93b12b4e5fbbf966
I20260926 15:00:45.135264 16644 ts_tablet_manager.cc:1939] T ef23f078457d411d93b12b4e5fbbf966 P 7aad28146b09442eace3bcf19cc525ea: Deleting consensus metadata
I20260926 15:00:45.135807 16096 catalog_manager.cc:4841] TS 7aad28146b09442eace3bcf19cc525ea (127.8.102.2:45043): tablet ef23f078457d411d93b12b4e5fbbf966 (table test-table [id=078a2ef3026f4d6e91d5820c34250cc6]) successfully deleted
I20260926 15:00:45.205976 16627 raft_consensus.cc:482] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:45.206092 16627 raft_consensus.cc:504] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.206537 16627 leader_election.cc:288] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7aad28146b09442eace3bcf19cc525ea (127.8.102.2:45043), efb7c3b9725645e786dc0797eb8fbfe6 (127.8.102.3:44483)
I20260926 15:00:45.210282 16616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.206985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:42599 (local address 127.8.102.2:45043)
0926 15:00:45.207027 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:45.207028 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:45.207123 (+ 95us) server_negotiation.cc:407] Connection header received
0926 15:00:45.207204 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:45.207205 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:45.207214 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:45.207279 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:45.207696 (+ 417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.208357 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.209411 (+ 1054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.209857 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.209929 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.209941 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:45.210133 (+ 192us) server_negotiation.cc:299] Negotiation successful
0926 15:00:45.210178 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 15:00:45.210309 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.206756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:45043 (local address 127.8.102.1:42599)
0926 15:00:45.207070 (+ 314us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:45.207079 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:45.207145 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:45.207328 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:45.207330 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:45.207332 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:45.207598 (+ 266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:45.207601 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.208473 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.208474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:45.209315 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:45.209317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.209967 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.209968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:45.210050 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.210055 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:00:45.210153 (+ 98us) client_negotiation.cc:240] Negotiation successful
0926 15:00:45.210203 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":300,"mutex_wait_us":2,"thread_start_us":88,"threads_started":1}
I20260926 15:00:45.210748 16618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.206973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:44019 (local address 127.8.102.3:44483)
0926 15:00:45.207026 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:45.207028 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:45.207071 (+ 43us) server_negotiation.cc:407] Connection header received
0926 15:00:45.207138 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:45.207139 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:45.207149 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:45.207208 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:45.207371 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.208231 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.209698 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.210205 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.210270 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.210282 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:45.210594 (+ 312us) server_negotiation.cc:299] Negotiation successful
0926 15:00:45.210675 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 15:00:45.210695 16406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6926963e08804af89f5997aa409e2496" candidate_uuid: "1cceaf11e17e46ee9180be9ec63ac941" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aad28146b09442eace3bcf19cc525ea" is_pre_election: true
I20260926 15:00:45.210824 16406 raft_consensus.cc:2454] T 6926963e08804af89f5997aa409e2496 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cceaf11e17e46ee9180be9ec63ac941 in term 0.
I20260926 15:00:45.210984 16650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.206756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:44483 (local address 127.8.102.1:44019)
0926 15:00:45.206989 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:45.207002 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:45.207057 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:45.207230 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:45.207231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:45.207233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:45.207311 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:45.207313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.208342 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.208344 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:45.209604 (+ 1260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:45.209607 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.210317 (+ 710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.210319 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:45.210505 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.210511 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 15:00:45.210826 (+ 315us) client_negotiation.cc:240] Negotiation successful
0926 15:00:45.210885 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"thread_start_us":59,"threads_started":1}
I20260926 15:00:45.211057 16200 leader_election.cc:302] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cceaf11e17e46ee9180be9ec63ac941, 7aad28146b09442eace3bcf19cc525ea; no voters:
I20260926 15:00:45.211200 16627 raft_consensus.cc:2781] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:00:45.211241 16627 raft_consensus.cc:482] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:00:45.211267 16627 raft_consensus.cc:3037] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:45.211426 16546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6926963e08804af89f5997aa409e2496" candidate_uuid: "1cceaf11e17e46ee9180be9ec63ac941" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" is_pre_election: true
I20260926 15:00:45.211512 16546 raft_consensus.cc:2454] T 6926963e08804af89f5997aa409e2496 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cceaf11e17e46ee9180be9ec63ac941 in term 0.
I20260926 15:00:45.212060 16627 raft_consensus.cc:504] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.212196 16627 leader_election.cc:288] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [CANDIDATE]: Term 1 election: Requested vote from peers 7aad28146b09442eace3bcf19cc525ea (127.8.102.2:45043), efb7c3b9725645e786dc0797eb8fbfe6 (127.8.102.3:44483)
I20260926 15:00:45.212344 16546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6926963e08804af89f5997aa409e2496" candidate_uuid: "1cceaf11e17e46ee9180be9ec63ac941" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb7c3b9725645e786dc0797eb8fbfe6"
I20260926 15:00:45.212453 16546 raft_consensus.cc:3037] T 6926963e08804af89f5997aa409e2496 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:45.213056 16546 raft_consensus.cc:2454] T 6926963e08804af89f5997aa409e2496 P efb7c3b9725645e786dc0797eb8fbfe6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cceaf11e17e46ee9180be9ec63ac941 in term 1.
I20260926 15:00:45.213207 16406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6926963e08804af89f5997aa409e2496" candidate_uuid: "1cceaf11e17e46ee9180be9ec63ac941" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aad28146b09442eace3bcf19cc525ea"
I20260926 15:00:45.213270 16198 leader_election.cc:302] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cceaf11e17e46ee9180be9ec63ac941, efb7c3b9725645e786dc0797eb8fbfe6; no voters:
I20260926 15:00:45.213301 16406 raft_consensus.cc:3037] T 6926963e08804af89f5997aa409e2496 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:45.213349 16627 raft_consensus.cc:2781] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:45.213497 16627 raft_consensus.cc:686] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [term 1 LEADER]: Becoming Leader. State: Replica: 1cceaf11e17e46ee9180be9ec63ac941, State: Running, Role: LEADER
I20260926 15:00:45.213572 16627 consensus_queue.cc:237] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.214110 16406 raft_consensus.cc:2454] T 6926963e08804af89f5997aa409e2496 P 7aad28146b09442eace3bcf19cc525ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cceaf11e17e46ee9180be9ec63ac941 in term 1.
I20260926 15:00:45.214277 16108 catalog_manager.cc:5495] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cceaf11e17e46ee9180be9ec63ac941 (127.8.102.1). New cstate: current_term: 1 leader_uuid: "1cceaf11e17e46ee9180be9ec63ac941" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } health_report { overall_health: UNKNOWN } } }
Sep 26 15:00:45 dist-test-slave-jnhs krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434844, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.62@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM
I20260926 15:00:45.250682 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.243377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45815 (local address 127.0.0.1:60510)
0926 15:00:45.243533 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:45.243543 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:45.243606 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:45.243743 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:45.243743 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:45.244062 (+ 319us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:45.244112 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:45.244114 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.244874 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.244875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:45.245526 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:45.245528 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.245873 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.245874 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:45.245941 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.246499 (+ 558us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:45.246503 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:45.248036 (+ 1533us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:45.249533 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:45.249540 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:45.249542 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:45.249733 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:45.249971 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:45.249972 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:45.249972 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:45.250039 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:45.250288 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:45.250295 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:45.250382 (+ 87us) client_negotiation.cc:769] Sending connection context
0926 15:00:45.250445 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 15:00:45.250550 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260926 15:00:45.250689 16174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.243438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60510 (local address 127.8.102.62:45815)
0926 15:00:45.243582 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:45.243584 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:45.243595 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:00:45.243665 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:45.243666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:45.243675 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:45.243728 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:45.244198 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.244762 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.245609 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.245796 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.245851 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.248118 (+ 2267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:45.248119 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:45.248119 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:45.248125 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:45.249435 (+ 1310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:45.249818 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:45.249819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:45.249820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:45.249878 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:45.250114 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:45.250114 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:45.250115 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:45.250224 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:45.250278 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:45.250485 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 15:00:45.250605 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":48,"server-negotiator.queue_time_us":113}
I20260926 15:00:45.252832 16108 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:60510:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260926 15:00:45.253039 16108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:00:45.254709 16246 tablet_service.cc:1467] Processing CreateTablet for tablet faf178245f1f4b548902dedc79a5a31c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ebdbeacecd5f46bfad696a5d5634d264]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:00:45.254719 16386 tablet_service.cc:1467] Processing CreateTablet for tablet faf178245f1f4b548902dedc79a5a31c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ebdbeacecd5f46bfad696a5d5634d264]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:00:45.254711 16526 tablet_service.cc:1467] Processing CreateTablet for tablet faf178245f1f4b548902dedc79a5a31c (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=ebdbeacecd5f46bfad696a5d5634d264]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260926 15:00:45.254881 16386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf178245f1f4b548902dedc79a5a31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:45.254881 16526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf178245f1f4b548902dedc79a5a31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:45.254894 16246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet faf178245f1f4b548902dedc79a5a31c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:45.256189 16621 tablet_bootstrap.cc:492] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941: Bootstrap starting.
I20260926 15:00:45.256662 16621 tablet_bootstrap.cc:654] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:45.257187 16620 tablet_bootstrap.cc:492] T faf178245f1f4b548902dedc79a5a31c P efb7c3b9725645e786dc0797eb8fbfe6: Bootstrap starting.
I20260926 15:00:45.257337 16621 tablet_bootstrap.cc:492] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941: No bootstrap required, opened a new log
I20260926 15:00:45.257376 16621 ts_tablet_manager.cc:1397] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:45.257480 16621 raft_consensus.cc:348] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.257509 16620 tablet_bootstrap.cc:654] T faf178245f1f4b548902dedc79a5a31c P efb7c3b9725645e786dc0797eb8fbfe6: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:45.257526 16621 raft_consensus.cc:374] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:45.257541 16621 raft_consensus.cc:729] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cceaf11e17e46ee9180be9ec63ac941, State: Initialized, Role: FOLLOWER
I20260926 15:00:45.257608 16621 consensus_queue.cc:260] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.257718 16621 ts_tablet_manager.cc:1428] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:45.257782 16652 tablet_replica.cc:439] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } } }
I20260926 15:00:45.257884 16652 tablet_replica.cc:442] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:00:45.258100 16620 tablet_bootstrap.cc:492] T faf178245f1f4b548902dedc79a5a31c P efb7c3b9725645e786dc0797eb8fbfe6: No bootstrap required, opened a new log
I20260926 15:00:45.258167 16620 ts_tablet_manager.cc:1397] T faf178245f1f4b548902dedc79a5a31c P efb7c3b9725645e786dc0797eb8fbfe6: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:45.258337 16622 tablet_bootstrap.cc:492] T faf178245f1f4b548902dedc79a5a31c P 7aad28146b09442eace3bcf19cc525ea: Bootstrap starting.
I20260926 15:00:45.258324 16620 raft_consensus.cc:348] T faf178245f1f4b548902dedc79a5a31c P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.258402 16620 raft_consensus.cc:374] T faf178245f1f4b548902dedc79a5a31c P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:45.258424 16620 raft_consensus.cc:729] T faf178245f1f4b548902dedc79a5a31c P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb7c3b9725645e786dc0797eb8fbfe6, State: Initialized, Role: FOLLOWER
I20260926 15:00:45.258463 16620 consensus_queue.cc:260] T faf178245f1f4b548902dedc79a5a31c P efb7c3b9725645e786dc0797eb8fbfe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.258560 16620 ts_tablet_manager.cc:1428] T faf178245f1f4b548902dedc79a5a31c P efb7c3b9725645e786dc0797eb8fbfe6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:45.258706 16622 tablet_bootstrap.cc:654] T faf178245f1f4b548902dedc79a5a31c P 7aad28146b09442eace3bcf19cc525ea: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:45.258618 16626 tablet_replica.cc:439] T faf178245f1f4b548902dedc79a5a31c P efb7c3b9725645e786dc0797eb8fbfe6: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } } }
I20260926 15:00:45.258752 16626 tablet_replica.cc:442] T faf178245f1f4b548902dedc79a5a31c P efb7c3b9725645e786dc0797eb8fbfe6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:00:45.259413 16622 tablet_bootstrap.cc:492] T faf178245f1f4b548902dedc79a5a31c P 7aad28146b09442eace3bcf19cc525ea: No bootstrap required, opened a new log
I20260926 15:00:45.259459 16622 ts_tablet_manager.cc:1397] T faf178245f1f4b548902dedc79a5a31c P 7aad28146b09442eace3bcf19cc525ea: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:45.259580 16622 raft_consensus.cc:348] T faf178245f1f4b548902dedc79a5a31c P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.259656 16622 raft_consensus.cc:374] T faf178245f1f4b548902dedc79a5a31c P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:45.259694 16622 raft_consensus.cc:729] T faf178245f1f4b548902dedc79a5a31c P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aad28146b09442eace3bcf19cc525ea, State: Initialized, Role: FOLLOWER
I20260926 15:00:45.259747 16622 consensus_queue.cc:260] T faf178245f1f4b548902dedc79a5a31c P 7aad28146b09442eace3bcf19cc525ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.259904 16622 ts_tablet_manager.cc:1428] T faf178245f1f4b548902dedc79a5a31c P 7aad28146b09442eace3bcf19cc525ea: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:45.259938 16640 tablet_replica.cc:439] T faf178245f1f4b548902dedc79a5a31c P 7aad28146b09442eace3bcf19cc525ea: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } } }
I20260926 15:00:45.260083 16640 tablet_replica.cc:442] T faf178245f1f4b548902dedc79a5a31c P 7aad28146b09442eace3bcf19cc525ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:00:45.615213 16652 raft_consensus.cc:482] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:45.615325 16652 raft_consensus.cc:504] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.615527 16652 leader_election.cc:288] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7aad28146b09442eace3bcf19cc525ea (127.8.102.2:45043), efb7c3b9725645e786dc0797eb8fbfe6 (127.8.102.3:44483)
I20260926 15:00:45.615772 16406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf178245f1f4b548902dedc79a5a31c" candidate_uuid: "1cceaf11e17e46ee9180be9ec63ac941" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aad28146b09442eace3bcf19cc525ea" is_pre_election: true
I20260926 15:00:45.615808 16546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf178245f1f4b548902dedc79a5a31c" candidate_uuid: "1cceaf11e17e46ee9180be9ec63ac941" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" is_pre_election: true
I20260926 15:00:45.615883 16546 raft_consensus.cc:2454] T faf178245f1f4b548902dedc79a5a31c P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cceaf11e17e46ee9180be9ec63ac941 in term 0.
I20260926 15:00:45.615876 16406 raft_consensus.cc:2454] T faf178245f1f4b548902dedc79a5a31c P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1cceaf11e17e46ee9180be9ec63ac941 in term 0.
I20260926 15:00:45.616037 16200 leader_election.cc:302] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cceaf11e17e46ee9180be9ec63ac941, 7aad28146b09442eace3bcf19cc525ea; no voters:
I20260926 15:00:45.616142 16652 raft_consensus.cc:2781] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:00:45.616178 16652 raft_consensus.cc:482] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:00:45.616201 16652 raft_consensus.cc:3037] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:45.616936 16652 raft_consensus.cc:504] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.617077 16652 leader_election.cc:288] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941 [CANDIDATE]: Term 1 election: Requested vote from peers 7aad28146b09442eace3bcf19cc525ea (127.8.102.2:45043), efb7c3b9725645e786dc0797eb8fbfe6 (127.8.102.3:44483)
I20260926 15:00:45.617256 16406 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf178245f1f4b548902dedc79a5a31c" candidate_uuid: "1cceaf11e17e46ee9180be9ec63ac941" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aad28146b09442eace3bcf19cc525ea"
I20260926 15:00:45.617327 16406 raft_consensus.cc:3037] T faf178245f1f4b548902dedc79a5a31c P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:45.617301 16546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "faf178245f1f4b548902dedc79a5a31c" candidate_uuid: "1cceaf11e17e46ee9180be9ec63ac941" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb7c3b9725645e786dc0797eb8fbfe6"
I20260926 15:00:45.617409 16546 raft_consensus.cc:3037] T faf178245f1f4b548902dedc79a5a31c P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:45.617993 16406 raft_consensus.cc:2454] T faf178245f1f4b548902dedc79a5a31c P 7aad28146b09442eace3bcf19cc525ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cceaf11e17e46ee9180be9ec63ac941 in term 1.
I20260926 15:00:45.618013 16546 raft_consensus.cc:2454] T faf178245f1f4b548902dedc79a5a31c P efb7c3b9725645e786dc0797eb8fbfe6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1cceaf11e17e46ee9180be9ec63ac941 in term 1.
I20260926 15:00:45.618188 16200 leader_election.cc:302] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cceaf11e17e46ee9180be9ec63ac941, 7aad28146b09442eace3bcf19cc525ea; no voters:
I20260926 15:00:45.618299 16652 raft_consensus.cc:2781] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:45.618355 16652 raft_consensus.cc:686] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941 [term 1 LEADER]: Becoming Leader. State: Replica: 1cceaf11e17e46ee9180be9ec63ac941, State: Running, Role: LEADER
I20260926 15:00:45.618391 16652 consensus_queue.cc:237] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.618708 16627 tablet_replica.cc:439] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941: TxnStatusTablet state changed. Reason: New leader 1cceaf11e17e46ee9180be9ec63ac941. Latest consensus state: current_term: 1 leader_uuid: "1cceaf11e17e46ee9180be9ec63ac941" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } } }
I20260926 15:00:45.618791 16627 tablet_replica.cc:442] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:00:45.618980 16664 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260926 15:00:45.619115 16664 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260926 15:00:45.619117 16108 catalog_manager.cc:5495] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1cceaf11e17e46ee9180be9ec63ac941 (127.8.102.1). New cstate: current_term: 1 leader_uuid: "1cceaf11e17e46ee9180be9ec63ac941" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } health_report { overall_health: UNKNOWN } } }
I20260926 15:00:45.699705 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.696913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:34405 (local address 127.0.0.1:38420)
0926 15:00:45.696988 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:45.697002 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:45.697071 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:45.697270 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:45.697271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:45.697272 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:00:45.697332 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:45.697334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.698042 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.698043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:45.698619 (+ 576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:45.698621 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.699109 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.699110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:45.699172 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.699187 (+ 15us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:00:45.699501 (+ 314us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:00:45.699512 (+ 11us) client_negotiation.cc:769] Sending connection context
0926 15:00:45.699567 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 15:00:45.699626 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 15:00:45.699713 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.696991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38420 (local address 127.8.102.1:34405)
0926 15:00:45.697095 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:45.697097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:45.697107 (+ 10us) server_negotiation.cc:407] Connection header received
0926 15:00:45.697146 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:45.697147 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:45.697169 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:45.697252 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:45.697404 (+ 152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.697956 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.698759 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.699034 (+ 275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.699111 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.699310 (+ 199us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:45.699413 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:45.699500 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:45.699573 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 15:00:45.699595 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":51,"threads_started":1}
I20260926 15:00:45.701112 16546 raft_consensus.cc:1261] T faf178245f1f4b548902dedc79a5a31c P efb7c3b9725645e786dc0797eb8fbfe6 [term 1 FOLLOWER]: Refusing update from remote peer 1cceaf11e17e46ee9180be9ec63ac941: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:45.701218 16406 raft_consensus.cc:1261] T faf178245f1f4b548902dedc79a5a31c P 7aad28146b09442eace3bcf19cc525ea [term 1 FOLLOWER]: Refusing update from remote peer 1cceaf11e17e46ee9180be9ec63ac941: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:45.701354 16652 consensus_queue.cc:1035] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:45.701536 16652 consensus_queue.cc:1035] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:45.701977 16626 tablet_replica.cc:439] T faf178245f1f4b548902dedc79a5a31c P efb7c3b9725645e786dc0797eb8fbfe6: TxnStatusTablet state changed. Reason: New leader 1cceaf11e17e46ee9180be9ec63ac941. Latest consensus state: current_term: 1 leader_uuid: "1cceaf11e17e46ee9180be9ec63ac941" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } } }
I20260926 15:00:45.702072 16626 tablet_replica.cc:442] T faf178245f1f4b548902dedc79a5a31c P efb7c3b9725645e786dc0797eb8fbfe6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:00:45.702265 16640 tablet_replica.cc:439] T faf178245f1f4b548902dedc79a5a31c P 7aad28146b09442eace3bcf19cc525ea: TxnStatusTablet state changed. Reason: New leader 1cceaf11e17e46ee9180be9ec63ac941. Latest consensus state: current_term: 1 leader_uuid: "1cceaf11e17e46ee9180be9ec63ac941" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } } }
I20260926 15:00:45.702350 16640 tablet_replica.cc:442] T faf178245f1f4b548902dedc79a5a31c P 7aad28146b09442eace3bcf19cc525ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:00:45.702708 16627 tablet_replica.cc:439] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cceaf11e17e46ee9180be9ec63ac941" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } } }
I20260926 15:00:45.702797 16627 tablet_replica.cc:442] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:00:45.703109 16652 tablet_replica.cc:439] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1cceaf11e17e46ee9180be9ec63ac941" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } } }
I20260926 15:00:45.703202 16652 tablet_replica.cc:442] T faf178245f1f4b548902dedc79a5a31c P 1cceaf11e17e46ee9180be9ec63ac941: This TxnStatusTablet replica's current role is: LEADER
I20260926 15:00:45.703583 16640 tablet_replica.cc:439] T faf178245f1f4b548902dedc79a5a31c P 7aad28146b09442eace3bcf19cc525ea: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cceaf11e17e46ee9180be9ec63ac941" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } } }
I20260926 15:00:45.703665 16640 tablet_replica.cc:442] T faf178245f1f4b548902dedc79a5a31c P 7aad28146b09442eace3bcf19cc525ea: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:00:45.703768 16671 mvcc.cc:205] Tried to move back new op lower bound from 7333621127989612544 to 7333621127653392384. Current Snapshot: MvccSnapshot[applied={T|T < 7333621127989612544}]
I20260926 15:00:45.707054 16626 tablet_replica.cc:439] T faf178245f1f4b548902dedc79a5a31c P efb7c3b9725645e786dc0797eb8fbfe6: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1cceaf11e17e46ee9180be9ec63ac941" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } } }
I20260926 15:00:45.707172 16626 tablet_replica.cc:442] T faf178245f1f4b548902dedc79a5a31c P efb7c3b9725645e786dc0797eb8fbfe6: This TxnStatusTablet replica's current role is: FOLLOWER
I20260926 15:00:45.711861 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.709280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38430 (local address 127.8.102.1:34405)
0926 15:00:45.709329 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:45.709331 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:45.709385 (+ 54us) server_negotiation.cc:407] Connection header received
0926 15:00:45.709459 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:45.709459 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:45.709471 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:45.709528 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:45.709677 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.710167 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.711113 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.711361 (+ 248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.711438 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.711608 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:45.711642 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:45.711690 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:45.711756 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 15:00:45.711777 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 15:00:45.717139 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.714298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:47941 (local address 127.8.102.1:34405)
0926 15:00:45.714341 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:45.714342 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:45.714513 (+ 171us) server_negotiation.cc:407] Connection header received
0926 15:00:45.714590 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:45.714591 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:45.714601 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:45.714711 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:45.714884 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.715349 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.716484 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.716658 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.716702 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.716892 (+ 190us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:45.716918 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:45.716960 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:45.717033 (+ 73us) server_negotiation.cc:299] Negotiation successful
0926 15:00:45.717050 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 15:00:45.717290 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.714244 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:34405 (local address 127.8.102.1:47941)
0926 15:00:45.714425 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:45.714437 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:45.714518 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:45.714718 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:45.714719 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:45.714721 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:00:45.714810 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:45.714811 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.715422 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.715423 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:45.716393 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:45.716396 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.716722 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.716723 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:45.716790 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.716808 (+ 18us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:00:45.716975 (+ 167us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:00:45.716978 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:00:45.717031 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 15:00:45.717080 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":74,"threads_started":1}
I20260926 15:00:45.719705 16546 raft_consensus.cc:1261] T 6926963e08804af89f5997aa409e2496 P efb7c3b9725645e786dc0797eb8fbfe6 [term 1 FOLLOWER]: Refusing update from remote peer 1cceaf11e17e46ee9180be9ec63ac941: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:45.719707 16406 raft_consensus.cc:1261] T 6926963e08804af89f5997aa409e2496 P 7aad28146b09442eace3bcf19cc525ea [term 1 FOLLOWER]: Refusing update from remote peer 1cceaf11e17e46ee9180be9ec63ac941: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:45.719925 16652 consensus_queue.cc:1035] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:45.720057 16674 consensus_queue.cc:1035] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:45.731803 16108 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:45.731927 16108 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60504:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:45.732537 16108 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260: Sending DeleteTablet for 3 replicas of tablet 6926963e08804af89f5997aa409e2496
I20260926 15:00:45.732825 16526 tablet_service.cc:1514] Processing DeleteTablet for tablet 6926963e08804af89f5997aa409e2496 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:45 UTC) from {username='oryx', principal='oryx/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:53326
I20260926 15:00:45.732868 16386 tablet_service.cc:1514] Processing DeleteTablet for tablet 6926963e08804af89f5997aa409e2496 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:45 UTC) from {username='oryx', principal='oryx/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:54986
I20260926 15:00:45.733002 16246 tablet_service.cc:1514] Processing DeleteTablet for tablet 6926963e08804af89f5997aa409e2496 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:45 UTC) from {username='oryx', principal='oryx/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:38408
I20260926 15:00:45.733166 16686 tablet_replica.cc:330] T 6926963e08804af89f5997aa409e2496 P 7aad28146b09442eace3bcf19cc525ea: stopping tablet replica
I20260926 15:00:45.733189 16685 tablet_replica.cc:330] T 6926963e08804af89f5997aa409e2496 P efb7c3b9725645e786dc0797eb8fbfe6: stopping tablet replica
I20260926 15:00:45.733242 16685 raft_consensus.cc:2229] T 6926963e08804af89f5997aa409e2496 P efb7c3b9725645e786dc0797eb8fbfe6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:45.733280 16686 raft_consensus.cc:2229] T 6926963e08804af89f5997aa409e2496 P 7aad28146b09442eace3bcf19cc525ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:45.733348 16686 raft_consensus.cc:2258] T 6926963e08804af89f5997aa409e2496 P 7aad28146b09442eace3bcf19cc525ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:45.733377 16685 raft_consensus.cc:2258] T 6926963e08804af89f5997aa409e2496 P efb7c3b9725645e786dc0797eb8fbfe6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:45.733675 16686 ts_tablet_manager.cc:1905] T 6926963e08804af89f5997aa409e2496 P 7aad28146b09442eace3bcf19cc525ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:45.733695 16685 ts_tablet_manager.cc:1905] T 6926963e08804af89f5997aa409e2496 P efb7c3b9725645e786dc0797eb8fbfe6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:45.735706 16685 ts_tablet_manager.cc:1918] T 6926963e08804af89f5997aa409e2496 P efb7c3b9725645e786dc0797eb8fbfe6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:00:45.735872 16685 log.cc:1196] T 6926963e08804af89f5997aa409e2496 P efb7c3b9725645e786dc0797eb8fbfe6: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/6926963e08804af89f5997aa409e2496
I20260926 15:00:45.735932 16686 ts_tablet_manager.cc:1918] T 6926963e08804af89f5997aa409e2496 P 7aad28146b09442eace3bcf19cc525ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:00:45.736091 16686 log.cc:1196] T 6926963e08804af89f5997aa409e2496 P 7aad28146b09442eace3bcf19cc525ea: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/6926963e08804af89f5997aa409e2496
I20260926 15:00:45.736187 16685 ts_tablet_manager.cc:1939] T 6926963e08804af89f5997aa409e2496 P efb7c3b9725645e786dc0797eb8fbfe6: Deleting consensus metadata
I20260926 15:00:45.736469 16686 ts_tablet_manager.cc:1939] T 6926963e08804af89f5997aa409e2496 P 7aad28146b09442eace3bcf19cc525ea: Deleting consensus metadata
I20260926 15:00:45.736617 16094 catalog_manager.cc:4841] TS efb7c3b9725645e786dc0797eb8fbfe6 (127.8.102.3:44483): tablet 6926963e08804af89f5997aa409e2496 (table test-table [id=2d7d5eda104d46c8bbd3b2ae341d49fe]) successfully deleted
I20260926 15:00:45.737085 16096 catalog_manager.cc:4841] TS 7aad28146b09442eace3bcf19cc525ea (127.8.102.2:45043): tablet 6926963e08804af89f5997aa409e2496 (table test-table [id=2d7d5eda104d46c8bbd3b2ae341d49fe]) successfully deleted
I20260926 15:00:45.737542 16687 tablet_replica.cc:330] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941: stopping tablet replica
I20260926 15:00:45.737630 16687 raft_consensus.cc:2229] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:00:45.737756 16687 raft_consensus.cc:2258] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:45.738029 16687 ts_tablet_manager.cc:1905] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:45.739560 16687 ts_tablet_manager.cc:1918] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:00:45.739643 16687 log.cc:1196] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-0/wal/wals/6926963e08804af89f5997aa409e2496
I20260926 15:00:45.739888 16687 ts_tablet_manager.cc:1939] T 6926963e08804af89f5997aa409e2496 P 1cceaf11e17e46ee9180be9ec63ac941: Deleting consensus metadata
I20260926 15:00:45.740249 16096 catalog_manager.cc:4841] TS 1cceaf11e17e46ee9180be9ec63ac941 (127.8.102.1:34405): tablet 6926963e08804af89f5997aa409e2496 (table test-table [id=2d7d5eda104d46c8bbd3b2ae341d49fe]) successfully deleted
Sep 26 15:00:45 dist-test-slave-jnhs krb5kdc[16064](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 26 15:00:45 dist-test-slave-jnhs krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:45 dist-test-slave-jnhs krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
W20260926 15:00:45.758641 16694 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:45.752803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60522 (local address 127.8.102.62:45815)
0926 15:00:45.752910 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:45.752913 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:45.753021 (+ 108us) server_negotiation.cc:407] Connection header received
0926 15:00:45.753126 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:45.753127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:45.753140 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:45.753208 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:45.753788 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.754302 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.755186 (+ 884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.755391 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.755490 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.758554 (+ 3064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60522: BlockingRecv error: recv got EOF from 127.0.0.1:60522 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":59,"threads_started":1}
Sep 26 15:00:45 dist-test-slave-jnhs krb5kdc[16064](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM
I20260926 15:00:45.767391 16694 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.759621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60532 (local address 127.8.102.62:45815)
0926 15:00:45.759662 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:45.759664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:45.759704 (+ 40us) server_negotiation.cc:407] Connection header received
0926 15:00:45.759787 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:45.759788 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:45.759797 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:45.759847 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:45.760428 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.760915 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.761619 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.761819 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.761904 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.764322 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:45.764324 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:45.764324 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:45.764332 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:45.766031 (+ 1699us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:45.766478 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:45.766479 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:45.766479 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:45.766547 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:45.766837 (+ 290us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:45.766838 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:45.766838 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:45.766990 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:45.767100 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:45.767206 (+ 106us) server_negotiation.cc:299] Negotiation successful
0926 15:00:45.767291 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 15:00:45.769114 16108 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:00:45.769279 16108 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:00:45.770694 16246 tablet_service.cc:1467] Processing CreateTablet for tablet 19b611b11f9041ea9b970537d0ad05a2 (DEFAULT_TABLE table=test-table [id=65084d9f03864fd285b933837eee0f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:45.770768 16386 tablet_service.cc:1467] Processing CreateTablet for tablet 19b611b11f9041ea9b970537d0ad05a2 (DEFAULT_TABLE table=test-table [id=65084d9f03864fd285b933837eee0f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:45.770784 16526 tablet_service.cc:1467] Processing CreateTablet for tablet 19b611b11f9041ea9b970537d0ad05a2 (DEFAULT_TABLE table=test-table [id=65084d9f03864fd285b933837eee0f11]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:45.770846 16246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b611b11f9041ea9b970537d0ad05a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:45.770900 16526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b611b11f9041ea9b970537d0ad05a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:45.770897 16386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 19b611b11f9041ea9b970537d0ad05a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:45.772360 16701 tablet_bootstrap.cc:492] T 19b611b11f9041ea9b970537d0ad05a2 P 1cceaf11e17e46ee9180be9ec63ac941: Bootstrap starting.
I20260926 15:00:45.772567 16702 tablet_bootstrap.cc:492] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6: Bootstrap starting.
I20260926 15:00:45.772770 16701 tablet_bootstrap.cc:654] T 19b611b11f9041ea9b970537d0ad05a2 P 1cceaf11e17e46ee9180be9ec63ac941: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:45.772902 16702 tablet_bootstrap.cc:654] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:45.773115 16703 tablet_bootstrap.cc:492] T 19b611b11f9041ea9b970537d0ad05a2 P 7aad28146b09442eace3bcf19cc525ea: Bootstrap starting.
I20260926 15:00:45.773489 16703 tablet_bootstrap.cc:654] T 19b611b11f9041ea9b970537d0ad05a2 P 7aad28146b09442eace3bcf19cc525ea: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:45.773494 16701 tablet_bootstrap.cc:492] T 19b611b11f9041ea9b970537d0ad05a2 P 1cceaf11e17e46ee9180be9ec63ac941: No bootstrap required, opened a new log
I20260926 15:00:45.773550 16701 ts_tablet_manager.cc:1397] T 19b611b11f9041ea9b970537d0ad05a2 P 1cceaf11e17e46ee9180be9ec63ac941: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:45.773674 16701 raft_consensus.cc:348] T 19b611b11f9041ea9b970537d0ad05a2 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.773765 16701 raft_consensus.cc:374] T 19b611b11f9041ea9b970537d0ad05a2 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:45.773797 16701 raft_consensus.cc:729] T 19b611b11f9041ea9b970537d0ad05a2 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cceaf11e17e46ee9180be9ec63ac941, State: Initialized, Role: FOLLOWER
I20260926 15:00:45.773854 16701 consensus_queue.cc:260] T 19b611b11f9041ea9b970537d0ad05a2 P 1cceaf11e17e46ee9180be9ec63ac941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.774021 16701 ts_tablet_manager.cc:1428] T 19b611b11f9041ea9b970537d0ad05a2 P 1cceaf11e17e46ee9180be9ec63ac941: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 15:00:45.774289 16703 tablet_bootstrap.cc:492] T 19b611b11f9041ea9b970537d0ad05a2 P 7aad28146b09442eace3bcf19cc525ea: No bootstrap required, opened a new log
I20260926 15:00:45.774364 16703 ts_tablet_manager.cc:1397] T 19b611b11f9041ea9b970537d0ad05a2 P 7aad28146b09442eace3bcf19cc525ea: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:45.774482 16703 raft_consensus.cc:348] T 19b611b11f9041ea9b970537d0ad05a2 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.774547 16703 raft_consensus.cc:374] T 19b611b11f9041ea9b970537d0ad05a2 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:45.774561 16703 raft_consensus.cc:729] T 19b611b11f9041ea9b970537d0ad05a2 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aad28146b09442eace3bcf19cc525ea, State: Initialized, Role: FOLLOWER
I20260926 15:00:45.774616 16703 consensus_queue.cc:260] T 19b611b11f9041ea9b970537d0ad05a2 P 7aad28146b09442eace3bcf19cc525ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.774772 16703 ts_tablet_manager.cc:1428] T 19b611b11f9041ea9b970537d0ad05a2 P 7aad28146b09442eace3bcf19cc525ea: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260926 15:00:45.775326 16702 tablet_bootstrap.cc:492] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6: No bootstrap required, opened a new log
I20260926 15:00:45.775401 16702 ts_tablet_manager.cc:1397] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260926 15:00:45.775561 16702 raft_consensus.cc:348] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.775629 16702 raft_consensus.cc:374] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:45.775643 16702 raft_consensus.cc:729] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb7c3b9725645e786dc0797eb8fbfe6, State: Initialized, Role: FOLLOWER
I20260926 15:00:45.775694 16702 consensus_queue.cc:260] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.775791 16702 ts_tablet_manager.cc:1428] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:45.989580 16626 raft_consensus.cc:482] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:45.989691 16626 raft_consensus.cc:504] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.990070 16626 leader_election.cc:288] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cceaf11e17e46ee9180be9ec63ac941 (127.8.102.1:34405), 7aad28146b09442eace3bcf19cc525ea (127.8.102.2:45043)
I20260926 15:00:45.993696 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.990505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:55945 (local address 127.8.102.1:34405)
0926 15:00:45.990586 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:45.990588 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:45.990704 (+ 116us) server_negotiation.cc:407] Connection header received
0926 15:00:45.990850 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:45.990852 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:45.990864 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:45.990922 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:45.991120 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.991687 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.992757 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.993332 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.993386 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.993402 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:45.993609 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 15:00:45.993639 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:00:45.993736 16707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.990336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:34405 (local address 127.8.102.3:55945)
0926 15:00:45.990566 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:45.990581 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:45.990720 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:45.990941 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:45.990942 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:45.990945 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:45.991031 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:45.991034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.991776 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.991778 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:45.992647 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:45.992650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.993414 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.993415 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:45.993531 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.993535 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:00:45.993599 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 15:00:45.993639 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":146,"threads_started":1}
I20260926 15:00:45.994117 16708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.990621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:45043 (local address 127.8.102.3:39535)
0926 15:00:45.990775 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:45.990790 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:45.990868 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:45.991047 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:45.991048 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:45.991051 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:00:45.991151 (+ 100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:45.991153 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.991922 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.991923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:45.993109 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:45.993112 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.993860 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.993861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:45.993938 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.993943 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:00:45.994002 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 15:00:45.994040 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":112,"threads_started":1}
I20260926 15:00:45.994079 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b611b11f9041ea9b970537d0ad05a2" candidate_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cceaf11e17e46ee9180be9ec63ac941" is_pre_election: true
I20260926 15:00:45.994177 16266 raft_consensus.cc:2454] T 19b611b11f9041ea9b970537d0ad05a2 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb7c3b9725645e786dc0797eb8fbfe6 in term 0.
I20260926 15:00:45.994338 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:45.990724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:39535 (local address 127.8.102.2:45043)
0926 15:00:45.990877 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:45.990878 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:45.990887 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:45.990944 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:45.990945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:45.990957 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:45.991023 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:00:45.991264 (+ 241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.991783 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.993204 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:45.993704 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:45.993814 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:45.993828 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:45.994205 (+ 377us) server_negotiation.cc:299] Negotiation successful
0926 15:00:45.994259 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":67,"threads_started":1}
I20260926 15:00:45.994360 16480 leader_election.cc:302] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cceaf11e17e46ee9180be9ec63ac941, efb7c3b9725645e786dc0797eb8fbfe6; no voters:
I20260926 15:00:45.994494 16626 raft_consensus.cc:2781] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:00:45.994529 16626 raft_consensus.cc:482] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:00:45.994549 16626 raft_consensus.cc:3037] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:45.994585 16405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b611b11f9041ea9b970537d0ad05a2" candidate_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aad28146b09442eace3bcf19cc525ea" is_pre_election: true
I20260926 15:00:45.994724 16405 raft_consensus.cc:2454] T 19b611b11f9041ea9b970537d0ad05a2 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate efb7c3b9725645e786dc0797eb8fbfe6 in term 0.
I20260926 15:00:45.995365 16626 raft_consensus.cc:504] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.995507 16626 leader_election.cc:288] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [CANDIDATE]: Term 1 election: Requested vote from peers 1cceaf11e17e46ee9180be9ec63ac941 (127.8.102.1:34405), 7aad28146b09442eace3bcf19cc525ea (127.8.102.2:45043)
I20260926 15:00:45.995630 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b611b11f9041ea9b970537d0ad05a2" candidate_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cceaf11e17e46ee9180be9ec63ac941"
I20260926 15:00:45.995662 16405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "19b611b11f9041ea9b970537d0ad05a2" candidate_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7aad28146b09442eace3bcf19cc525ea"
I20260926 15:00:45.995715 16266 raft_consensus.cc:3037] T 19b611b11f9041ea9b970537d0ad05a2 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:45.995751 16405 raft_consensus.cc:3037] T 19b611b11f9041ea9b970537d0ad05a2 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:45.996340 16266 raft_consensus.cc:2454] T 19b611b11f9041ea9b970537d0ad05a2 P 1cceaf11e17e46ee9180be9ec63ac941 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb7c3b9725645e786dc0797eb8fbfe6 in term 1.
I20260926 15:00:45.996423 16405 raft_consensus.cc:2454] T 19b611b11f9041ea9b970537d0ad05a2 P 7aad28146b09442eace3bcf19cc525ea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate efb7c3b9725645e786dc0797eb8fbfe6 in term 1.
I20260926 15:00:45.996611 16480 leader_election.cc:302] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7aad28146b09442eace3bcf19cc525ea, efb7c3b9725645e786dc0797eb8fbfe6; no voters:
I20260926 15:00:45.996706 16626 raft_consensus.cc:2781] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:45.996870 16626 raft_consensus.cc:686] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [term 1 LEADER]: Becoming Leader. State: Replica: efb7c3b9725645e786dc0797eb8fbfe6, State: Running, Role: LEADER
I20260926 15:00:45.996951 16626 consensus_queue.cc:237] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:45.997715 16109 catalog_manager.cc:5495] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 reported cstate change: term changed from 0 to 1, leader changed from <none> to efb7c3b9725645e786dc0797eb8fbfe6 (127.8.102.3). New cstate: current_term: 1 leader_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } health_report { overall_health: HEALTHY } } }
I20260926 15:00:46.002882 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:46.000256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53334 (local address 127.8.102.3:44483)
0926 15:00:46.000408 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:46.000409 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:46.000417 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:00:46.000456 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:46.000457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:46.000475 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:46.000548 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:46.000681 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:46.001166 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:46.001940 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:46.002180 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:46.002245 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:46.002454 (+ 209us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:46.002567 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:46.002699 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:46.002737 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 15:00:46.002763 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":104,"threads_started":1}
I20260926 15:00:46.003890 16266 raft_consensus.cc:1261] T 19b611b11f9041ea9b970537d0ad05a2 P 1cceaf11e17e46ee9180be9ec63ac941 [term 1 FOLLOWER]: Refusing update from remote peer efb7c3b9725645e786dc0797eb8fbfe6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:46.003890 16405 raft_consensus.cc:1261] T 19b611b11f9041ea9b970537d0ad05a2 P 7aad28146b09442eace3bcf19cc525ea [term 1 FOLLOWER]: Refusing update from remote peer efb7c3b9725645e786dc0797eb8fbfe6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:46.004249 16626 consensus_queue.cc:1035] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:46.004406 16710 consensus_queue.cc:1035] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:46.008098 16109 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:46.008213 16109 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:46.008852 16109 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260: Sending DeleteTablet for 3 replicas of tablet 19b611b11f9041ea9b970537d0ad05a2
I20260926 15:00:46.009117 16526 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b611b11f9041ea9b970537d0ad05a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:46 UTC) from {username='oryx', principal='oryx/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:53326
I20260926 15:00:46.009135 16246 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b611b11f9041ea9b970537d0ad05a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:46 UTC) from {username='oryx', principal='oryx/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:38408
I20260926 15:00:46.009205 16685 tablet_replica.cc:330] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6: stopping tablet replica
I20260926 15:00:46.009255 16685 raft_consensus.cc:2229] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:00:46.009246 16687 tablet_replica.cc:330] T 19b611b11f9041ea9b970537d0ad05a2 P 1cceaf11e17e46ee9180be9ec63ac941: stopping tablet replica
I20260926 15:00:46.009307 16687 raft_consensus.cc:2229] T 19b611b11f9041ea9b970537d0ad05a2 P 1cceaf11e17e46ee9180be9ec63ac941 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:46.009331 16685 raft_consensus.cc:2258] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:46.009351 16687 raft_consensus.cc:2258] T 19b611b11f9041ea9b970537d0ad05a2 P 1cceaf11e17e46ee9180be9ec63ac941 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:46.009429 16386 tablet_service.cc:1514] Processing DeleteTablet for tablet 19b611b11f9041ea9b970537d0ad05a2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:46 UTC) from {username='oryx', principal='oryx/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:54986
I20260926 15:00:46.009519 16686 tablet_replica.cc:330] T 19b611b11f9041ea9b970537d0ad05a2 P 7aad28146b09442eace3bcf19cc525ea: stopping tablet replica
I20260926 15:00:46.009593 16686 raft_consensus.cc:2229] T 19b611b11f9041ea9b970537d0ad05a2 P 7aad28146b09442eace3bcf19cc525ea [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:46.009649 16686 raft_consensus.cc:2258] T 19b611b11f9041ea9b970537d0ad05a2 P 7aad28146b09442eace3bcf19cc525ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:46.009723 16685 ts_tablet_manager.cc:1905] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:46.009737 16687 ts_tablet_manager.cc:1905] T 19b611b11f9041ea9b970537d0ad05a2 P 1cceaf11e17e46ee9180be9ec63ac941: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:46.010839 16109 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60532:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:00:46.011006 16109 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:00:46.011310 16687 ts_tablet_manager.cc:1918] T 19b611b11f9041ea9b970537d0ad05a2 P 1cceaf11e17e46ee9180be9ec63ac941: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:00:46.011430 16687 log.cc:1196] T 19b611b11f9041ea9b970537d0ad05a2 P 1cceaf11e17e46ee9180be9ec63ac941: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-0/wal/wals/19b611b11f9041ea9b970537d0ad05a2
I20260926 15:00:46.011427 16686 ts_tablet_manager.cc:1905] T 19b611b11f9041ea9b970537d0ad05a2 P 7aad28146b09442eace3bcf19cc525ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:46.011657 16687 ts_tablet_manager.cc:1939] T 19b611b11f9041ea9b970537d0ad05a2 P 1cceaf11e17e46ee9180be9ec63ac941: Deleting consensus metadata
I20260926 15:00:46.012476 16526 tablet_service.cc:1467] Processing CreateTablet for tablet ed8f75669101422c821e2ee8de90da40 (DEFAULT_TABLE table=test-table [id=77ced22dcbed411aae23c2c47f19089d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:46.012559 16685 ts_tablet_manager.cc:1918] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:00:46.012601 16526 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8f75669101422c821e2ee8de90da40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:46.012614 16685 log.cc:1196] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/19b611b11f9041ea9b970537d0ad05a2
I20260926 15:00:46.012847 16685 ts_tablet_manager.cc:1939] T 19b611b11f9041ea9b970537d0ad05a2 P efb7c3b9725645e786dc0797eb8fbfe6: Deleting consensus metadata
I20260926 15:00:46.013029 16686 ts_tablet_manager.cc:1918] T 19b611b11f9041ea9b970537d0ad05a2 P 7aad28146b09442eace3bcf19cc525ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:00:46.013080 16686 log.cc:1196] T 19b611b11f9041ea9b970537d0ad05a2 P 7aad28146b09442eace3bcf19cc525ea: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/19b611b11f9041ea9b970537d0ad05a2
I20260926 15:00:46.013150 16094 catalog_manager.cc:4841] TS efb7c3b9725645e786dc0797eb8fbfe6 (127.8.102.3:44483): tablet 19b611b11f9041ea9b970537d0ad05a2 (table test-table [id=65084d9f03864fd285b933837eee0f11]) successfully deleted
I20260926 15:00:46.013295 16686 ts_tablet_manager.cc:1939] T 19b611b11f9041ea9b970537d0ad05a2 P 7aad28146b09442eace3bcf19cc525ea: Deleting consensus metadata
I20260926 15:00:46.013545 16096 catalog_manager.cc:4841] TS 1cceaf11e17e46ee9180be9ec63ac941 (127.8.102.1:34405): tablet 19b611b11f9041ea9b970537d0ad05a2 (table test-table [id=65084d9f03864fd285b933837eee0f11]) successfully deleted
I20260926 15:00:46.013661 16096 catalog_manager.cc:4841] TS 7aad28146b09442eace3bcf19cc525ea (127.8.102.2:45043): tablet 19b611b11f9041ea9b970537d0ad05a2 (table test-table [id=65084d9f03864fd285b933837eee0f11]) successfully deleted
I20260926 15:00:46.013741 16386 tablet_service.cc:1467] Processing CreateTablet for tablet ed8f75669101422c821e2ee8de90da40 (DEFAULT_TABLE table=test-table [id=77ced22dcbed411aae23c2c47f19089d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:46.013847 16386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8f75669101422c821e2ee8de90da40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:46.013908 16702 tablet_bootstrap.cc:492] T ed8f75669101422c821e2ee8de90da40 P efb7c3b9725645e786dc0797eb8fbfe6: Bootstrap starting.
I20260926 15:00:46.014307 16702 tablet_bootstrap.cc:654] T ed8f75669101422c821e2ee8de90da40 P efb7c3b9725645e786dc0797eb8fbfe6: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:46.014899 16703 tablet_bootstrap.cc:492] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea: Bootstrap starting.
I20260926 15:00:46.014998 16246 tablet_service.cc:1467] Processing CreateTablet for tablet ed8f75669101422c821e2ee8de90da40 (DEFAULT_TABLE table=test-table [id=77ced22dcbed411aae23c2c47f19089d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:00:46.015093 16246 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ed8f75669101422c821e2ee8de90da40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:46.015183 16703 tablet_bootstrap.cc:654] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:46.015564 16702 tablet_bootstrap.cc:492] T ed8f75669101422c821e2ee8de90da40 P efb7c3b9725645e786dc0797eb8fbfe6: No bootstrap required, opened a new log
I20260926 15:00:46.015607 16702 ts_tablet_manager.cc:1397] T ed8f75669101422c821e2ee8de90da40 P efb7c3b9725645e786dc0797eb8fbfe6: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:00:46.015749 16702 raft_consensus.cc:348] T ed8f75669101422c821e2ee8de90da40 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:46.015811 16702 raft_consensus.cc:374] T ed8f75669101422c821e2ee8de90da40 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:46.015831 16702 raft_consensus.cc:729] T ed8f75669101422c821e2ee8de90da40 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efb7c3b9725645e786dc0797eb8fbfe6, State: Initialized, Role: FOLLOWER
I20260926 15:00:46.015877 16702 consensus_queue.cc:260] T ed8f75669101422c821e2ee8de90da40 P efb7c3b9725645e786dc0797eb8fbfe6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:46.015956 16702 ts_tablet_manager.cc:1428] T ed8f75669101422c821e2ee8de90da40 P efb7c3b9725645e786dc0797eb8fbfe6: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:46.016237 16701 tablet_bootstrap.cc:492] T ed8f75669101422c821e2ee8de90da40 P 1cceaf11e17e46ee9180be9ec63ac941: Bootstrap starting.
I20260926 15:00:46.016346 16703 tablet_bootstrap.cc:492] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea: No bootstrap required, opened a new log
I20260926 15:00:46.016399 16703 ts_tablet_manager.cc:1397] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:00:46.016501 16703 raft_consensus.cc:348] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:46.016551 16701 tablet_bootstrap.cc:654] T ed8f75669101422c821e2ee8de90da40 P 1cceaf11e17e46ee9180be9ec63ac941: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:46.016571 16703 raft_consensus.cc:374] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:46.016594 16703 raft_consensus.cc:729] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7aad28146b09442eace3bcf19cc525ea, State: Initialized, Role: FOLLOWER
I20260926 15:00:46.016647 16703 consensus_queue.cc:260] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:46.016752 16703 ts_tablet_manager.cc:1428] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:46.017043 16701 tablet_bootstrap.cc:492] T ed8f75669101422c821e2ee8de90da40 P 1cceaf11e17e46ee9180be9ec63ac941: No bootstrap required, opened a new log
I20260926 15:00:46.017083 16701 ts_tablet_manager.cc:1397] T ed8f75669101422c821e2ee8de90da40 P 1cceaf11e17e46ee9180be9ec63ac941: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:46.017189 16701 raft_consensus.cc:348] T ed8f75669101422c821e2ee8de90da40 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:46.017242 16701 raft_consensus.cc:374] T ed8f75669101422c821e2ee8de90da40 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:46.017268 16701 raft_consensus.cc:729] T ed8f75669101422c821e2ee8de90da40 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cceaf11e17e46ee9180be9ec63ac941, State: Initialized, Role: FOLLOWER
I20260926 15:00:46.017310 16701 consensus_queue.cc:260] T ed8f75669101422c821e2ee8de90da40 P 1cceaf11e17e46ee9180be9ec63ac941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:46.017416 16701 ts_tablet_manager.cc:1428] T ed8f75669101422c821e2ee8de90da40 P 1cceaf11e17e46ee9180be9ec63ac941: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:46.200230 16640 raft_consensus.cc:482] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:00:46.200371 16640 raft_consensus.cc:504] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:46.200580 16640 leader_election.cc:288] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1cceaf11e17e46ee9180be9ec63ac941 (127.8.102.1:34405), efb7c3b9725645e786dc0797eb8fbfe6 (127.8.102.3:44483)
I20260926 15:00:46.200807 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed8f75669101422c821e2ee8de90da40" candidate_uuid: "7aad28146b09442eace3bcf19cc525ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cceaf11e17e46ee9180be9ec63ac941" is_pre_election: true
I20260926 15:00:46.200891 16266 raft_consensus.cc:2454] T ed8f75669101422c821e2ee8de90da40 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7aad28146b09442eace3bcf19cc525ea in term 0.
I20260926 15:00:46.200865 16546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed8f75669101422c821e2ee8de90da40" candidate_uuid: "7aad28146b09442eace3bcf19cc525ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" is_pre_election: true
I20260926 15:00:46.201009 16546 raft_consensus.cc:2454] T ed8f75669101422c821e2ee8de90da40 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7aad28146b09442eace3bcf19cc525ea in term 0.
I20260926 15:00:46.201077 16340 leader_election.cc:302] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cceaf11e17e46ee9180be9ec63ac941, 7aad28146b09442eace3bcf19cc525ea; no voters:
I20260926 15:00:46.201190 16640 raft_consensus.cc:2781] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:00:46.201246 16640 raft_consensus.cc:482] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:00:46.201261 16640 raft_consensus.cc:3037] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:46.202011 16640 raft_consensus.cc:504] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:46.202170 16640 leader_election.cc:288] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [CANDIDATE]: Term 1 election: Requested vote from peers 1cceaf11e17e46ee9180be9ec63ac941 (127.8.102.1:34405), efb7c3b9725645e786dc0797eb8fbfe6 (127.8.102.3:44483)
I20260926 15:00:46.202351 16266 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed8f75669101422c821e2ee8de90da40" candidate_uuid: "7aad28146b09442eace3bcf19cc525ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1cceaf11e17e46ee9180be9ec63ac941"
I20260926 15:00:46.202363 16546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ed8f75669101422c821e2ee8de90da40" candidate_uuid: "7aad28146b09442eace3bcf19cc525ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "efb7c3b9725645e786dc0797eb8fbfe6"
I20260926 15:00:46.202458 16546 raft_consensus.cc:3037] T ed8f75669101422c821e2ee8de90da40 P efb7c3b9725645e786dc0797eb8fbfe6 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:46.202463 16266 raft_consensus.cc:3037] T ed8f75669101422c821e2ee8de90da40 P 1cceaf11e17e46ee9180be9ec63ac941 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:46.203220 16266 raft_consensus.cc:2454] T ed8f75669101422c821e2ee8de90da40 P 1cceaf11e17e46ee9180be9ec63ac941 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7aad28146b09442eace3bcf19cc525ea in term 1.
I20260926 15:00:46.203441 16340 leader_election.cc:302] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1cceaf11e17e46ee9180be9ec63ac941, 7aad28146b09442eace3bcf19cc525ea; no voters:
I20260926 15:00:46.203495 16546 raft_consensus.cc:2454] T ed8f75669101422c821e2ee8de90da40 P efb7c3b9725645e786dc0797eb8fbfe6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7aad28146b09442eace3bcf19cc525ea in term 1.
I20260926 15:00:46.203540 16640 raft_consensus.cc:2781] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:46.203646 16640 raft_consensus.cc:686] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [term 1 LEADER]: Becoming Leader. State: Replica: 7aad28146b09442eace3bcf19cc525ea, State: Running, Role: LEADER
I20260926 15:00:46.203699 16640 consensus_queue.cc:237] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } }
I20260926 15:00:46.204435 16109 catalog_manager.cc:5495] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea reported cstate change: term changed from 0 to 1, leader changed from <none> to 7aad28146b09442eace3bcf19cc525ea (127.8.102.2). New cstate: current_term: 1 leader_uuid: "7aad28146b09442eace3bcf19cc525ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7aad28146b09442eace3bcf19cc525ea" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45043 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 } health_report { overall_health: UNKNOWN } } }
I20260926 15:00:46.246222 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:46.243928 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54996 (local address 127.8.102.2:45043)
0926 15:00:46.243995 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:46.243997 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:46.244006 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:46.244085 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:46.244086 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:46.244096 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:46.244142 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:46.244235 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:46.244756 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:46.245577 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:46.245794 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:46.245866 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:46.245987 (+ 121us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:46.246016 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:46.246060 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:46.246088 (+ 28us) server_negotiation.cc:299] Negotiation successful
0926 15:00:46.246110 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 15:00:46.251240 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:46.248996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:42353 (local address 127.8.102.1:34405)
0926 15:00:46.249082 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:46.249084 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:46.249142 (+ 58us) server_negotiation.cc:407] Connection header received
0926 15:00:46.249182 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:46.249183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:46.249195 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:46.249244 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:46.249370 (+ 126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:46.249832 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:46.250599 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:46.250852 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:46.250907 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:46.251057 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:46.251083 (+ 26us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:46.251118 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:46.251158 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 15:00:46.251179 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:00:46.251255 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:46.248917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:34405 (local address 127.8.102.2:42353)
0926 15:00:46.249061 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:46.249075 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:46.249137 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:46.249254 (+ 117us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:46.249254 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:46.249256 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:00:46.249296 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:46.249297 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:46.249922 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:46.249923 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:46.250519 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:46.250521 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:46.250916 (+ 395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:46.250917 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:46.250961 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:46.250975 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:00:46.251126 (+ 151us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:00:46.251127 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 15:00:46.251157 (+ 30us) client_negotiation.cc:240] Negotiation successful
0926 15:00:46.251194 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":76,"threads_started":1}
I20260926 15:00:46.253485 16266 raft_consensus.cc:1261] T ed8f75669101422c821e2ee8de90da40 P 1cceaf11e17e46ee9180be9ec63ac941 [term 1 FOLLOWER]: Refusing update from remote peer 7aad28146b09442eace3bcf19cc525ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:46.253717 16546 raft_consensus.cc:1261] T ed8f75669101422c821e2ee8de90da40 P efb7c3b9725645e786dc0797eb8fbfe6 [term 1 FOLLOWER]: Refusing update from remote peer 7aad28146b09442eace3bcf19cc525ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:00:46.253708 16640 consensus_queue.cc:1035] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "1cceaf11e17e46ee9180be9ec63ac941" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34405 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:46.253923 16640 consensus_queue.cc:1035] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "efb7c3b9725645e786dc0797eb8fbfe6" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:00:46.260967 16731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:46.258354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:45043 (local address 127.8.102.1:38697)
0926 15:00:46.258509 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:46.258525 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:46.258595 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:46.258738 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:46.258739 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:46.258740 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0926 15:00:46.258787 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:46.258789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:46.259616 (+ 827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:46.259617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:46.260212 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:46.260214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:46.260622 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:46.260623 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:46.260684 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:46.260689 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0926 15:00:46.260824 (+ 135us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0926 15:00:46.260825 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 15:00:46.260856 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 15:00:46.260896 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":103,"threads_started":1}
I20260926 15:00:46.260933 16709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:46.258388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:38697 (local address 127.8.102.2:45043)
0926 15:00:46.258442 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:46.258444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:46.258578 (+ 134us) server_negotiation.cc:407] Connection header received
0926 15:00:46.258668 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:46.258669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:46.258680 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:46.258734 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:00:46.258835 (+ 101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:46.259525 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:46.260303 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:46.260522 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:46.260607 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:46.260764 (+ 157us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:00:46.260789 (+ 25us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:00:46.260823 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:46.260857 (+ 34us) server_negotiation.cc:299] Negotiation successful
0926 15:00:46.260878 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 15:00:46.267608 16108 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:46.267741 16108 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60532:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:00:46.268461 16108 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5c4bd5ffb5c475ebb429d7e50fe3260: Sending DeleteTablet for 3 replicas of tablet ed8f75669101422c821e2ee8de90da40
I20260926 15:00:46.268723 16526 tablet_service.cc:1514] Processing DeleteTablet for tablet ed8f75669101422c821e2ee8de90da40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:46 UTC) from {username='oryx', principal='oryx/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:53326
I20260926 15:00:46.268748 16386 tablet_service.cc:1514] Processing DeleteTablet for tablet ed8f75669101422c821e2ee8de90da40 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:00:46 UTC) from {username='oryx', principal='oryx/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:54986
I20260926 15:00:46.268823 16685 tablet_replica.cc:330] T ed8f75669101422c821e2ee8de90da40 P efb7c3b9725645e786dc0797eb8fbfe6: stopping tablet replica
I20260926 15:00:46.268874 16685 raft_consensus.cc:2229] T ed8f75669101422c821e2ee8de90da40 P efb7c3b9725645e786dc0797eb8fbfe6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:00:46.268910 16685 raft_consensus.cc:2258] T ed8f75669101422c821e2ee8de90da40 P efb7c3b9725645e786dc0797eb8fbfe6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:46.269021 16686 tablet_replica.cc:330] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea: stopping tablet replica
I20260926 15:00:46.269025 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 16181
I20260926 15:00:46.269084 16686 raft_consensus.cc:2229] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:00:46.269150 16686 raft_consensus.cc:2258] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:00:46.270069 16685 ts_tablet_manager.cc:1905] T ed8f75669101422c821e2ee8de90da40 P efb7c3b9725645e786dc0797eb8fbfe6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:46.271991 16685 ts_tablet_manager.cc:1918] T ed8f75669101422c821e2ee8de90da40 P efb7c3b9725645e786dc0797eb8fbfe6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:00:46.272084 16685 log.cc:1196] T ed8f75669101422c821e2ee8de90da40 P efb7c3b9725645e786dc0797eb8fbfe6: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/ed8f75669101422c821e2ee8de90da40
I20260926 15:00:46.272298 16685 ts_tablet_manager.cc:1939] T ed8f75669101422c821e2ee8de90da40 P efb7c3b9725645e786dc0797eb8fbfe6: Deleting consensus metadata
I20260926 15:00:46.272612 16686 ts_tablet_manager.cc:1905] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:00:46.273943 16094 catalog_manager.cc:4841] TS efb7c3b9725645e786dc0797eb8fbfe6 (127.8.102.3:44483): tablet ed8f75669101422c821e2ee8de90da40 (table test-table [id=77ced22dcbed411aae23c2c47f19089d]) successfully deleted
I20260926 15:00:46.274200 16686 ts_tablet_manager.cc:1918] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260926 15:00:46.274266 16686 log.cc:1196] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/ed8f75669101422c821e2ee8de90da40
I20260926 15:00:46.274570 16686 ts_tablet_manager.cc:1939] T ed8f75669101422c821e2ee8de90da40 P 7aad28146b09442eace3bcf19cc525ea: Deleting consensus metadata
I20260926 15:00:46.274976 16096 catalog_manager.cc:4841] TS 7aad28146b09442eace3bcf19cc525ea (127.8.102.2:45043): tablet ed8f75669101422c821e2ee8de90da40 (table test-table [id=77ced22dcbed411aae23c2c47f19089d]) successfully deleted
W20260926 15:00:46.276831 16096 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.102.1:34405 (error 108)
I20260926 15:00:46.277199 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 16321
I20260926 15:00:46.277444 16734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:46.277185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:34405 (local address 127.0.0.1:38438)
0926 15:00:46.277335 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:46.277395 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.1:34405: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":50,"threads_started":1}
W20260926 15:00:46.277822 16096 catalog_manager.cc:4584] TS 1cceaf11e17e46ee9180be9ec63ac941 (127.8.102.1:34405): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ed8f75669101422c821e2ee8de90da40: Network error: Client connection negotiation failed: client connection to 127.8.102.1:34405: connect: Connection refused (error 111)
I20260926 15:00:46.283620 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 16461
I20260926 15:00:46.289690 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 16077
2026-09-26T15:00:46Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (3511 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:00:46 dist-test-slave-jnhs krb5kdc[16737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:00:46 dist-test-slave-jnhs krb5kdc[16737](info): set up 2 sockets
Sep 26 15:00:46 dist-test-slave-jnhs krb5kdc[16737](info): commencing operation
krb5kdc: starting...
W20260926 15:00:47.327430 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:00:47 dist-test-slave-jnhs krb5kdc[16737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434847, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:00:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:00:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:00:47.454836 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:45617
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:43403
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:45617
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:47.547410 16750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:47.547648 16750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:47.547672 16750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:47.548918 16750 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:47.548970 16750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:47.548985 16750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:47.548998 16750 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:47.549010 16750 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:00:47.550627 16750 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:43403
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:45617
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:45617
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16750
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:47.550927 16750 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:47.551177 16750 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:47.556737 16758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:47.556859 16755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:47.556969 16756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:47.557336 16750 server_base.cc:1034] running on GCE node
I20260926 15:00:47.557653 16750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:47.557971 16750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:47.559149 16750 hybrid_clock.cc:648] HybridClock initialized: now 1790434847559125 us; error 41 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-26T15:00:47Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1282 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:00:47 dist-test-slave-jnhs krb5kdc[16763](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:00:47 dist-test-slave-jnhs krb5kdc[16763](info): set up 2 sockets
Sep 26 15:00:47 dist-test-slave-jnhs krb5kdc[16763](info): commencing operation
krb5kdc: starting...
W20260926 15:00:48.643406 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.050s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:00:48 dist-test-slave-jnhs krb5kdc[16763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:00:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:00:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:00:48.775173 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:41841
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:41841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:48.857645 16776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:48.857863 16776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:48.857887 16776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:48.859117 16776 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:48.859166 16776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:48.859181 16776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:48.859193 16776 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:48.859206 16776 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:00:48.860850 16776 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:41841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:41841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:48.861119 16776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:48.861390 16776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:48.866456 16782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:48.866456 16781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:48.866515 16776 server_base.cc:1034] running on GCE node
W20260926 15:00:48.866743 16784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:48.867013 16776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:48.867274 16776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:48.868438 16776 hybrid_clock.cc:648] HybridClock initialized: now 1790434848868426 us; error 30 us; skew 500 ppm
Sep 26 15:00:48 dist-test-slave-jnhs krb5kdc[16763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434848, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:48.870980 16776 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:00:48.871491 16776 webserver.cc:460] Webserver started at http://127.8.102.62:36595/ using document root <none> and password file <none>
I20260926 15:00:48.871750 16776 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:48.871797 16776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:48.871990 16776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:48.873062 16776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "f2a124b15a944474a96eb382644f55a3"
format_stamp: "Formatted at 2026-09-26 15:00:48 on dist-test-slave-jnhs"
server_key: "da0f75d2dc3e0ca41a84639b9afce0ff"
server_key_iv: "77ca7d5b56bba3a6e2206e9c1da83ddf"
server_key_version: "encryptionkey@0"
I20260926 15:00:48.873417 16776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "f2a124b15a944474a96eb382644f55a3"
format_stamp: "Formatted at 2026-09-26 15:00:48 on dist-test-slave-jnhs"
server_key: "da0f75d2dc3e0ca41a84639b9afce0ff"
server_key_iv: "77ca7d5b56bba3a6e2206e9c1da83ddf"
server_key_version: "encryptionkey@0"
I20260926 15:00:48.874878 16776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:00:48.875634 16791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:48.875809 16776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 15:00:48.875928 16776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "f2a124b15a944474a96eb382644f55a3"
format_stamp: "Formatted at 2026-09-26 15:00:48 on dist-test-slave-jnhs"
server_key: "da0f75d2dc3e0ca41a84639b9afce0ff"
server_key_iv: "77ca7d5b56bba3a6e2206e9c1da83ddf"
server_key_version: "encryptionkey@0"
I20260926 15:00:48.876001 16776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:48.889838 16776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:48.892205 16776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:48.892365 16776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:48.897001 16776 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:41841
I20260926 15:00:48.897038 16843 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:41841 every 8 connection(s)
I20260926 15:00:48.897445 16776 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:00:48.898363 16844 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:00:48.900295 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 16776
I20260926 15:00:48.900388 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:00:48.900645 8600 external_mini_cluster.cc:1428] Setting key f0255ff8f614268e30ae49b1b0d6cad5
I20260926 15:00:48.900935 16844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3: Bootstrap starting.
I20260926 15:00:48.901669 16844 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3: Neither blocks nor log segments found. Creating new log.
I20260926 15:00:48.901960 16844 log.cc:824] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:48.902822 16844 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3: No bootstrap required, opened a new log
I20260926 15:00:48.904901 16844 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a124b15a944474a96eb382644f55a3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41841 } }
I20260926 15:00:48.905112 16844 raft_consensus.cc:374] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:00:48.905143 16844 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a124b15a944474a96eb382644f55a3, State: Initialized, Role: FOLLOWER
I20260926 15:00:48.905296 16844 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a124b15a944474a96eb382644f55a3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41841 } }
I20260926 15:00:48.905372 16844 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:00:48.905416 16844 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:00:48.905462 16844 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:00:48.906243 16844 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a124b15a944474a96eb382644f55a3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41841 } }
I20260926 15:00:48.906364 16844 leader_election.cc:302] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2a124b15a944474a96eb382644f55a3; no voters:
I20260926 15:00:48.906539 16844 leader_election.cc:288] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:00:48.906751 16850 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:00:48.906855 16844 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:48.906905 16850 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [term 1 LEADER]: Becoming Leader. State: Replica: f2a124b15a944474a96eb382644f55a3, State: Running, Role: LEADER
I20260926 15:00:48.907094 16850 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a124b15a944474a96eb382644f55a3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41841 } }
I20260926 15:00:48.907505 16850 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2a124b15a944474a96eb382644f55a3. Latest consensus state: current_term: 1 leader_uuid: "f2a124b15a944474a96eb382644f55a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a124b15a944474a96eb382644f55a3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41841 } } }
I20260926 15:00:48.907522 16851 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2a124b15a944474a96eb382644f55a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a124b15a944474a96eb382644f55a3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41841 } } }
I20260926 15:00:48.907670 16850 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:48.907678 16851 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:48.907919 16858 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 26 15:00:48 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434848, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:48.908603 16858 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:00:48.910370 16858 catalog_manager.cc:1347] Generated new cluster ID: 4387bf65007b40c296c80f790d0bb711
I20260926 15:00:48.910423 16858 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:00:48.912846 16847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:48.901221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49190 (local address 127.8.102.62:41841)
0926 15:00:48.901406 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:48.901414 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:48.901440 (+ 26us) server_negotiation.cc:407] Connection header received
0926 15:00:48.901909 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:48.901916 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:48.901964 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:48.902145 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:48.902968 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:48.903898 (+ 930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:48.905054 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:48.905702 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:48.905793 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:48.908882 (+ 3089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:48.908886 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:48.908886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:48.908898 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:48.911100 (+ 2202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:48.911504 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:48.911507 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:48.911507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:48.911622 (+ 115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:48.911825 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:48.911826 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:48.911826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:48.912158 (+ 332us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:48.912323 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:48.912465 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 15:00:48.912619 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":100,"threads_started":1}
I20260926 15:00:48.926761 16858 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:00:48.927565 16858 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:00:48.931785 16858 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3: Generated new TSK 0
I20260926 15:00:48.932030 16858 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
I20260926 15:00:48.963963 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:41841
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:49.047492 16870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:49.047708 16870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:49.047732 16870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:49.049006 16870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:49.049049 16870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:49.049101 16870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:00:49.050557 16870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:41841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16870
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:49.050837 16870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:49.051091 16870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:49.056084 16876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:49.056216 16878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:49.056108 16870 server_base.cc:1034] running on GCE node
W20260926 15:00:49.056093 16875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:49.056555 16870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:49.056824 16870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:49.057996 16870 hybrid_clock.cc:648] HybridClock initialized: now 1790434849057992 us; error 35 us; skew 500 ppm
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434849, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:49.060611 16870 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:00:49.061050 16870 webserver.cc:460] Webserver started at http://127.8.102.1:43957/ using document root <none> and password file <none>
I20260926 15:00:49.061283 16870 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:49.061350 16870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:49.061543 16870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:49.062624 16870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "b860711d8ee94c39b7969dc49d3b2b42"
format_stamp: "Formatted at 2026-09-26 15:00:49 on dist-test-slave-jnhs"
server_key: "0fb350c1d0a742cb116d2b0fab21cad3"
server_key_iv: "735d185e53b8831e3cfb79411629e631"
server_key_version: "encryptionkey@0"
I20260926 15:00:49.063071 16870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "b860711d8ee94c39b7969dc49d3b2b42"
format_stamp: "Formatted at 2026-09-26 15:00:49 on dist-test-slave-jnhs"
server_key: "0fb350c1d0a742cb116d2b0fab21cad3"
server_key_iv: "735d185e53b8831e3cfb79411629e631"
server_key_version: "encryptionkey@0"
I20260926 15:00:49.064391 16870 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:00:49.065155 16885 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:49.065353 16870 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 15:00:49.065516 16870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "b860711d8ee94c39b7969dc49d3b2b42"
format_stamp: "Formatted at 2026-09-26 15:00:49 on dist-test-slave-jnhs"
server_key: "0fb350c1d0a742cb116d2b0fab21cad3"
server_key_iv: "735d185e53b8831e3cfb79411629e631"
server_key_version: "encryptionkey@0"
I20260926 15:00:49.065605 16870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:49.093619 16870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:49.095690 16870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:49.095857 16870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:49.096107 16870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:49.096447 16870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:49.096500 16870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:49.096536 16870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:49.096562 16870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:49.102442 16870 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:39929
I20260926 15:00:49.102484 16998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:39929 every 8 connection(s)
I20260926 15:00:49.102855 16870 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:00:49.109311 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 16870
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434849, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:49.109405 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:00:49.109609 8600 external_mini_cluster.cc:1428] Setting key 25997aebfa8d68e13b470125810be0f9
I20260926 15:00:49.113191 16847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:49.103634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:37157 (local address 127.8.102.62:41841)
0926 15:00:49.103704 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:49.103707 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:49.103911 (+ 204us) server_negotiation.cc:407] Connection header received
0926 15:00:49.104426 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:49.104428 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:49.104440 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:49.104521 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:49.105491 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.106003 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.107187 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.107526 (+ 339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.107622 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:49.109950 (+ 2328us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:49.109951 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:49.109952 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:49.109958 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:49.111469 (+ 1511us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:49.111919 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:49.111921 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:49.111921 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:49.111978 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:49.112235 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:49.112236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:49.112236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:49.112355 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:49.112437 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:49.112901 (+ 464us) server_negotiation.cc:299] Negotiation successful
0926 15:00:49.113064 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 15:00:49.113201 17001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:49.103598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.1:37157)
0926 15:00:49.103774 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:49.103818 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:49.104316 (+ 498us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:49.104581 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:49.104584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:49.105179 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:49.105389 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:49.105394 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.106084 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.106084 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:49.107086 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:49.107090 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.107635 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.107636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:49.107762 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:49.108281 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:49.108289 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:49.109843 (+ 1554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:49.111574 (+ 1731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:49.111581 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:49.111584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:49.111819 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:49.112046 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:49.112047 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:49.112047 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:49.112166 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:49.112427 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:49.112433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:49.112603 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 15:00:49.112716 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 15:00:49.112914 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":101,"threads_started":1}
I20260926 15:00:49.113842 16999 heartbeater.cc:347] Connected to a master server at 127.8.102.62:41841
I20260926 15:00:49.113952 16999 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:49.114140 16999 heartbeater.cc:511] Master 127.8.102.62:41841 requested a full tablet report, sending...
I20260926 15:00:49.114766 16808 ts_manager.cc:194] Registered new tserver with Master: b860711d8ee94c39b7969dc49d3b2b42 (127.8.102.1:39929)
I20260926 15:00:49.115428 16808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:37157
WARNING: no policy specified for kudu/127.8.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
I20260926 15:00:49.138391 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:41841
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:49.220417 17004 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:49.220687 17004 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:49.220727 17004 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:49.221941 17004 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:49.222026 17004 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:49.222076 17004 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:00:49.223634 17004 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:41841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17004
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:49.223920 17004 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:49.224172 17004 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:49.229571 17009 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:49.229581 17012 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:49.229737 17004 server_base.cc:1034] running on GCE node
W20260926 15:00:49.229585 17010 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:49.229964 17004 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:49.230266 17004 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:49.231459 17004 hybrid_clock.cc:648] HybridClock initialized: now 1790434849231437 us; error 39 us; skew 500 ppm
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434849, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:49.234251 17004 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:00:49.234803 17004 webserver.cc:460] Webserver started at http://127.8.102.2:34307/ using document root <none> and password file <none>
I20260926 15:00:49.235049 17004 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:49.235116 17004 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:49.235256 17004 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:49.236362 17004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "2547f7c5caf54fbd847ef94ed216e475"
format_stamp: "Formatted at 2026-09-26 15:00:49 on dist-test-slave-jnhs"
server_key: "1ea0222120e56ae9807fc2bb5fb3d252"
server_key_iv: "fd2722cfdb28a9148aed8dd43fdaa704"
server_key_version: "encryptionkey@0"
I20260926 15:00:49.236756 17004 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "2547f7c5caf54fbd847ef94ed216e475"
format_stamp: "Formatted at 2026-09-26 15:00:49 on dist-test-slave-jnhs"
server_key: "1ea0222120e56ae9807fc2bb5fb3d252"
server_key_iv: "fd2722cfdb28a9148aed8dd43fdaa704"
server_key_version: "encryptionkey@0"
I20260926 15:00:49.238251 17004 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:00:49.239385 17019 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:49.239589 17004 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 15:00:49.239696 17004 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "2547f7c5caf54fbd847ef94ed216e475"
format_stamp: "Formatted at 2026-09-26 15:00:49 on dist-test-slave-jnhs"
server_key: "1ea0222120e56ae9807fc2bb5fb3d252"
server_key_iv: "fd2722cfdb28a9148aed8dd43fdaa704"
server_key_version: "encryptionkey@0"
I20260926 15:00:49.239789 17004 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:49.247337 17004 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:49.249558 17004 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:49.249740 17004 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:49.249992 17004 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:49.250310 17004 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:49.250367 17004 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:49.250429 17004 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:49.250473 17004 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:49.257368 17004 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:46485
I20260926 15:00:49.257416 17132 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:46485 every 8 connection(s)
I20260926 15:00:49.257795 17004 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:00:49.263562 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 17004
I20260926 15:00:49.263700 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:00:49.263937 8600 external_mini_cluster.cc:1428] Setting key 348a080b0acf40c3aa55e8917599f878
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434849, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:49.268306 16847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:49.258615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:57471 (local address 127.8.102.62:41841)
0926 15:00:49.258729 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:49.258732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:49.258922 (+ 190us) server_negotiation.cc:407] Connection header received
0926 15:00:49.259450 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:49.259450 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:49.259461 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:49.259548 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:49.260521 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.261054 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.262111 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.262312 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.262377 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:49.265185 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:49.265188 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:49.265188 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:49.265197 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:49.266724 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:49.267268 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:49.267269 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:49.267270 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:49.267333 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:49.267624 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:49.267625 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:49.267625 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:49.267753 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:49.267834 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:49.268116 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 15:00:49.268210 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260926 15:00:49.268527 17135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:49.258581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.2:57471)
0926 15:00:49.258806 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:49.258846 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:49.259353 (+ 507us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:49.259605 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:49.259609 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:49.260197 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:49.260402 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:49.260407 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.261149 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.261150 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:49.262014 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:49.262016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.262408 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.262408 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:49.262532 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:49.263232 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:49.263241 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:49.265091 (+ 1850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:49.266800 (+ 1709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:49.266808 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:49.266811 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:49.267170 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:49.267406 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:49.267407 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:49.267407 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:49.267536 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:49.267824 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:49.267830 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:49.267975 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 15:00:49.268095 (+ 120us) client_negotiation.cc:240] Negotiation successful
0926 15:00:49.268270 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":181,"thread_start_us":133,"threads_started":1}
I20260926 15:00:49.269141 17133 heartbeater.cc:347] Connected to a master server at 127.8.102.62:41841
I20260926 15:00:49.269297 17133 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:49.269550 17133 heartbeater.cc:511] Master 127.8.102.62:41841 requested a full tablet report, sending...
I20260926 15:00:49.270077 16808 ts_manager.cc:194] Registered new tserver with Master: 2547f7c5caf54fbd847ef94ed216e475 (127.8.102.2:46485)
I20260926 15:00:49.270627 16808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:57471
WARNING: no policy specified for kudu/127.8.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
I20260926 15:00:49.294714 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:41841
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:49.384239 17138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:49.384454 17138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:49.384478 17138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:49.385655 17138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:49.385697 17138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:49.385725 17138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:00:49.387220 17138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:41841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17138
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:49.387475 17138 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:49.387710 17138 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:49.393011 17143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:49.392988 17144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:49.392990 17146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:49.393239 17138 server_base.cc:1034] running on GCE node
I20260926 15:00:49.393554 17138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:49.393775 17138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:49.395015 17138 hybrid_clock.cc:648] HybridClock initialized: now 1790434849394999 us; error 46 us; skew 500 ppm
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434849, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:49.397665 17138 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:00:49.398209 17138 webserver.cc:460] Webserver started at http://127.8.102.3:36315/ using document root <none> and password file <none>
I20260926 15:00:49.398478 17138 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:49.398550 17138 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:49.398730 17138 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:00:49.399889 17138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "da79b95282a8406899b34479fefb98f3"
format_stamp: "Formatted at 2026-09-26 15:00:49 on dist-test-slave-jnhs"
server_key: "3bc408d5c0474ee2967e70568fb9e99d"
server_key_iv: "aad91932fcf80f080e4ca135aba499ef"
server_key_version: "encryptionkey@0"
I20260926 15:00:49.400264 17138 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "da79b95282a8406899b34479fefb98f3"
format_stamp: "Formatted at 2026-09-26 15:00:49 on dist-test-slave-jnhs"
server_key: "3bc408d5c0474ee2967e70568fb9e99d"
server_key_iv: "aad91932fcf80f080e4ca135aba499ef"
server_key_version: "encryptionkey@0"
I20260926 15:00:49.401681 17138 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:00:49.402479 17153 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:49.402717 17138 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:49.402827 17138 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "da79b95282a8406899b34479fefb98f3"
format_stamp: "Formatted at 2026-09-26 15:00:49 on dist-test-slave-jnhs"
server_key: "3bc408d5c0474ee2967e70568fb9e99d"
server_key_iv: "aad91932fcf80f080e4ca135aba499ef"
server_key_version: "encryptionkey@0"
I20260926 15:00:49.402907 17138 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:49.414722 17138 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:49.417128 17138 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:49.417289 17138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:49.417543 17138 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:49.417869 17138 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:49.417925 17138 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:49.417968 17138 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:49.417997 17138 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:49.423924 17138 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:33963
I20260926 15:00:49.423986 17266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:33963 every 8 connection(s)
I20260926 15:00:49.424357 17138 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:00:49.430258 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 17138
I20260926 15:00:49.430341 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:00:49.430624 8600 external_mini_cluster.cc:1428] Setting key 11ee22ffea6d64c8bc545a7ca593c3b7
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434849, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:00:49.434180 16847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:49.425151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:33435 (local address 127.8.102.62:41841)
0926 15:00:49.425218 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:49.425220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:49.425429 (+ 209us) server_negotiation.cc:407] Connection header received
0926 15:00:49.426066 (+ 637us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:49.426067 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:49.426080 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:49.426125 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:49.427184 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.427728 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.428775 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.428975 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.429080 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:49.431648 (+ 2568us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:49.431650 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:49.431650 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:49.431659 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:49.432922 (+ 1263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:49.433298 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:49.433299 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:49.433299 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:49.433350 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:49.433552 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:49.433552 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:49.433553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:49.433657 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:49.433708 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:49.433991 (+ 283us) server_negotiation.cc:299] Negotiation successful
0926 15:00:49.434080 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 15:00:49.434401 17269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:49.425132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.3:33435)
0926 15:00:49.425306 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:49.425343 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:49.425954 (+ 611us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:49.426199 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:49.426203 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:49.426848 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:49.427083 (+ 235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:49.427089 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.427832 (+ 743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.427833 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:49.428666 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:49.428669 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.429088 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.429089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:49.429223 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:49.429768 (+ 545us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:49.429777 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:49.431555 (+ 1778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:00:49.432998 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:49.433005 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:49.433007 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:49.433231 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:49.433391 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:00:49.433392 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:00:49.433392 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:00:49.433492 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:00:49.433713 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:00:49.433718 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:00:49.433839 (+ 121us) client_negotiation.cc:769] Sending connection context
0926 15:00:49.433987 (+ 148us) client_negotiation.cc:240] Negotiation successful
0926 15:00:49.434159 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":86,"threads_started":1}
I20260926 15:00:49.434903 17267 heartbeater.cc:347] Connected to a master server at 127.8.102.62:41841
I20260926 15:00:49.435006 17267 heartbeater.cc:464] Registering TS with master...
I20260926 15:00:49.435230 17267 heartbeater.cc:511] Master 127.8.102.62:41841 requested a full tablet report, sending...
I20260926 15:00:49.435636 16808 ts_manager.cc:194] Registered new tserver with Master: da79b95282a8406899b34479fefb98f3 (127.8.102.3:33963)
I20260926 15:00:49.436034 16808 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:33435
I20260926 15:00:49.442968 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.1.keytab.
Entry for principal oryx/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.1.keytab.
WARNING: no policy specified for oryx/127.8.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.2.keytab.
Entry for principal oryx/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.2.keytab.
WARNING: no policy specified for oryx/127.8.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.3.keytab.
Entry for principal oryx/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.3.keytab.
I20260926 15:00:49.526597 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 16870
I20260926 15:00:49.531769 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 17004
I20260926 15:00:49.536466 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 17138
I20260926 15:00:49.540740 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 16776
I20260926 15:00:49.546023 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:41841
--webserver_interface=127.8.102.62
--webserver_port=36595
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:41841
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:00:49.630266 17276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:49.630545 17276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:49.630581 17276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:49.631784 17276 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:00:49.631866 17276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:49.631891 17276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:49.631924 17276 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:00:49.631971 17276 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:00:49.633479 17276 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:41841
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:41841
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=36595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17276
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:49.633771 17276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:49.634033 17276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:49.639070 17281 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:49.639071 17282 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:49.639065 17284 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:49.639309 17276 server_base.cc:1034] running on GCE node
I20260926 15:00:49.639689 17276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:49.640015 17276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:49.641197 17276 hybrid_clock.cc:648] HybridClock initialized: now 1790434849641174 us; error 40 us; skew 500 ppm
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434849, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:49.643837 17276 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:00:49.644335 17276 webserver.cc:460] Webserver started at http://127.8.102.62:36595/ using document root <none> and password file <none>
I20260926 15:00:49.644593 17276 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:49.644654 17276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:49.646026 17276 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:49.646862 17291 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:49.647037 17276 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:49.647142 17276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "f2a124b15a944474a96eb382644f55a3"
format_stamp: "Formatted at 2026-09-26 15:00:48 on dist-test-slave-jnhs"
server_key: "da0f75d2dc3e0ca41a84639b9afce0ff"
server_key_iv: "77ca7d5b56bba3a6e2206e9c1da83ddf"
server_key_version: "encryptionkey@0"
I20260926 15:00:49.647521 17276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:49.692965 17276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:49.695446 17276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:49.695616 17276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:49.699604 17276 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:41841
I20260926 15:00:49.699635 17343 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:41841 every 8 connection(s)
I20260926 15:00:49.700035 17276 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:00:49.701586 17344 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:00:49.701651 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 17276
I20260926 15:00:49.701951 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:39929
--local_ip_for_outbound_sockets=127.8.102.1
--tserver_master_addrs=127.8.102.62:41841
--webserver_port=43957
--webserver_interface=127.8.102.1
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.102.1
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.1.keytab with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:00:49.702653 17344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3: Bootstrap starting.
I20260926 15:00:49.710953 17344 log.cc:824] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3: Log is configured to *not* fsync() on all Append() calls
I20260926 15:00:49.712752 17344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:00:49.713070 17344 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3: Bootstrap complete.
I20260926 15:00:49.715698 17344 raft_consensus.cc:348] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a124b15a944474a96eb382644f55a3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41841 } }
I20260926 15:00:49.715924 17344 raft_consensus.cc:729] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a124b15a944474a96eb382644f55a3, State: Initialized, Role: FOLLOWER
I20260926 15:00:49.716089 17344 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a124b15a944474a96eb382644f55a3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41841 } }
I20260926 15:00:49.716164 17344 raft_consensus.cc:388] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:00:49.716228 17344 raft_consensus.cc:482] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:00:49.716272 17344 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [term 1 FOLLOWER]: Advancing to term 2
I20260926 15:00:49.717334 17344 raft_consensus.cc:504] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a124b15a944474a96eb382644f55a3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41841 } }
I20260926 15:00:49.717453 17344 leader_election.cc:302] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2a124b15a944474a96eb382644f55a3; no voters:
I20260926 15:00:49.717633 17344 leader_election.cc:288] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 15:00:49.717705 17348 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 15:00:49.717911 17344 sys_catalog.cc:564] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:00:49.717927 17348 raft_consensus.cc:686] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [term 2 LEADER]: Becoming Leader. State: Replica: f2a124b15a944474a96eb382644f55a3, State: Running, Role: LEADER
I20260926 15:00:49.718158 17348 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a124b15a944474a96eb382644f55a3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41841 } }
I20260926 15:00:49.718446 17349 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "f2a124b15a944474a96eb382644f55a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a124b15a944474a96eb382644f55a3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41841 } } }
I20260926 15:00:49.718513 17349 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:49.718806 17350 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2a124b15a944474a96eb382644f55a3. Latest consensus state: current_term: 2 leader_uuid: "f2a124b15a944474a96eb382644f55a3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a124b15a944474a96eb382644f55a3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41841 } } }
I20260926 15:00:49.718863 17350 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3 [sys.catalog]: This master's current role is: LEADER
I20260926 15:00:49.719015 17355 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:00:49.719558 17355 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:00:49.719736 17355 catalog_manager.cc:1259] Loaded cluster ID: 4387bf65007b40c296c80f790d0bb711
I20260926 15:00:49.719955 17355 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:00:49.721230 17355 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:00:49.721624 17355 catalog_manager.cc:5879] T 00000000000000000000000000000000 P f2a124b15a944474a96eb382644f55a3: Loaded TSK: 0
I20260926 15:00:49.721912 17355 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 15:00:49.786413 17346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:49.786717 17346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:49.786787 17346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:49.788038 17346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:49.788127 17346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:49.788177 17346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:00:49.789691 17346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.1.keytab
--principal=oryx/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:39929
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=43957
--tserver_master_addrs=127.8.102.62:41841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:49.789968 17346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:49.790208 17346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:49.796357 17371 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:49.796370 17372 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:49.796625 17346 server_base.cc:1034] running on GCE node
W20260926 15:00:49.796379 17374 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:49.796955 17346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:49.797250 17346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:49.798499 17346 hybrid_clock.cc:648] HybridClock initialized: now 1790434849798483 us; error 37 us; skew 500 ppm
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434849, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:49.801872 17346 init.cc:377] Logged in from keytab as oryx/127.8.102.1@KRBTEST.COM (short username oryx)
I20260926 15:00:49.802421 17346 webserver.cc:460] Webserver started at http://127.8.102.1:43957/ using document root <none> and password file <none>
I20260926 15:00:49.802732 17346 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:49.802796 17346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:49.804306 17346 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:49.805114 17381 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:49.805276 17346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:49.805379 17346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "b860711d8ee94c39b7969dc49d3b2b42"
format_stamp: "Formatted at 2026-09-26 15:00:49 on dist-test-slave-jnhs"
server_key: "0fb350c1d0a742cb116d2b0fab21cad3"
server_key_iv: "735d185e53b8831e3cfb79411629e631"
server_key_version: "encryptionkey@0"
I20260926 15:00:49.805781 17346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:49.815578 17346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:49.817976 17346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:49.818122 17346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:49.818391 17346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:49.818809 17346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:49.818845 17346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:49.818867 17346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:49.818881 17346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:49.825119 17346 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:39929
I20260926 15:00:49.825160 17494 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:39929 every 8 connection(s)
I20260926 15:00:49.825505 17346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:00:49.827123 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 17346
I20260926 15:00:49.827421 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:46485
--local_ip_for_outbound_sockets=127.8.102.2
--tserver_master_addrs=127.8.102.62:41841
--webserver_port=34307
--webserver_interface=127.8.102.2
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.102.2
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.2.keytab with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:49.836760 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:49.826210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.1:35111)
0926 15:00:49.826434 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:49.826485 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:49.827179 (+ 694us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:49.827801 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:49.827805 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:49.828594 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:49.828858 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:49.828864 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.830249 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.830250 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:49.831680 (+ 1430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:49.831683 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.832620 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.832621 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:49.832764 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:49.833436 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:49.833446 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:49.836494 (+ 3048us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":99,"threads_started":1}
W20260926 15:00:49.837023 17498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:49.826339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:35111 (local address 127.8.102.62:41841)
0926 15:00:49.826768 (+ 429us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:49.826775 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:49.826804 (+ 29us) server_negotiation.cc:407] Connection header received
0926 15:00:49.827459 (+ 655us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:49.827466 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:49.827522 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:49.827652 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:49.828997 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.830134 (+ 1137us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.831784 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.832492 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.832635 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:49.836670 (+ 4035us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.1:35111: BlockingRecv error: recv got EOF from 127.8.102.1:35111 (error 108)
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":79,"threads_started":1}
W20260926 15:00:49.837122 17495 heartbeater.cc:644] Failed to heartbeat to 127.8.102.62:41841 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.102.62:41841: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:49.842825 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:49.837340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.1:46723)
0926 15:00:49.837395 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:49.837407 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:49.837447 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:49.837600 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:49.837601 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:49.837768 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:49.837832 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:49.837833 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.838739 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.838740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:49.839655 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:49.839658 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.840126 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.840127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:49.840192 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:49.840680 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:49.840685 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:49.842751 (+ 2066us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 15:00:49.843004 17498 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:49.837403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:46723 (local address 127.8.102.62:41841)
0926 15:00:49.837435 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:49.837436 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:49.837445 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:49.837516 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:49.837517 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:49.837528 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:49.837578 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:49.837914 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.838669 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.839751 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.840046 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.840117 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:49.842853 (+ 2736us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.1:46723: BlockingRecv error: recv got EOF from 127.8.102.1:46723 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:49.848724 17497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:49.843175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.1:41821)
0926 15:00:49.843227 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:49.843239 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:49.843278 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:49.843538 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:49.843539 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:49.843684 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:49.843720 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:49.843722 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.844667 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.844668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:49.845482 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:49.845486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.846009 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.846010 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:49.846078 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:49.846589 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:49.846594 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:49.848622 (+ 2028us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 15:00:49.848965 17495 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 15:00:49.849093 17500 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:49.843226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:41821 (local address 127.8.102.62:41841)
0926 15:00:49.843398 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:49.843400 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:49.843406 (+ 6us) server_negotiation.cc:407] Connection header received
0926 15:00:49.843429 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:49.843430 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:49.843441 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:49.843502 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:49.843801 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.844573 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.845583 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.845917 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.845996 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:49.848760 (+ 2764us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.1:41821: BlockingRecv error: recv got EOF from 127.8.102.1:41821 (error 108)
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":108,"threads_started":1}
W20260926 15:00:49.916879 17499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:49.917104 17499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:49.917129 17499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:49.918376 17499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:49.918426 17499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:49.918454 17499 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:00:49.920174 17499 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.2.keytab
--principal=oryx/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:46485
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=34307
--tserver_master_addrs=127.8.102.62:41841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17499
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:49.920401 17499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:49.920585 17499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:49.925936 17506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:49.925926 17505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:49.926016 17499 server_base.cc:1034] running on GCE node
W20260926 15:00:49.925926 17508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:49.926433 17499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:49.926786 17499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:49.927942 17499 hybrid_clock.cc:648] HybridClock initialized: now 1790434849927932 us; error 31 us; skew 500 ppm
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434849, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:49.930625 17499 init.cc:377] Logged in from keytab as oryx/127.8.102.2@KRBTEST.COM (short username oryx)
I20260926 15:00:49.931213 17499 webserver.cc:460] Webserver started at http://127.8.102.2:34307/ using document root <none> and password file <none>
I20260926 15:00:49.931463 17499 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:49.931550 17499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:49.932994 17499 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:49.933706 17515 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:49.933902 17499 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:49.934022 17499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "2547f7c5caf54fbd847ef94ed216e475"
format_stamp: "Formatted at 2026-09-26 15:00:49 on dist-test-slave-jnhs"
server_key: "1ea0222120e56ae9807fc2bb5fb3d252"
server_key_iv: "fd2722cfdb28a9148aed8dd43fdaa704"
server_key_version: "encryptionkey@0"
I20260926 15:00:49.934386 17499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:49.950910 17499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:49.953367 17499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:49.953531 17499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:49.953763 17499 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:49.954083 17499 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:49.954135 17499 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:49.954176 17499 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:49.954205 17499 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:49.960350 17499 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:46485
I20260926 15:00:49.960402 17628 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:46485 every 8 connection(s)
I20260926 15:00:49.960781 17499 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:00:49.963543 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 17499
I20260926 15:00:49.963850 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:33963
--local_ip_for_outbound_sockets=127.8.102.3
--tserver_master_addrs=127.8.102.62:41841
--webserver_port=36315
--webserver_interface=127.8.102.3
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:34453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.8.102.3
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.3.keytab with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:49.969399 17631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:49.961566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.2:49185)
0926 15:00:49.961724 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:49.961761 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:49.962255 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:49.962505 (+ 250us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:49.962509 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:49.963164 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:49.963421 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:49.963429 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.964289 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.964290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:49.965532 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:49.965536 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.966089 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.966090 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:49.966197 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:49.966788 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:49.966797 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:49.969243 (+ 2446us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":94,"threads_started":1}
W20260926 15:00:49.969465 17500 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:49.961570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:49185 (local address 127.8.102.62:41841)
0926 15:00:49.961646 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:49.961648 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:49.961835 (+ 187us) server_negotiation.cc:407] Connection header received
0926 15:00:49.962366 (+ 531us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:49.962367 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:49.962383 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:49.962453 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:49.963536 (+ 1083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.964194 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.965651 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.965981 (+ 330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.966065 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:49.969379 (+ 3314us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.2:49185: BlockingRecv error: recv got EOF from 127.8.102.2:49185 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
W20260926 15:00:49.969753 17629 heartbeater.cc:644] Failed to heartbeat to 127.8.102.62:41841 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.102.62:41841: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:49.975001 17631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:49.970011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.2:44025)
0926 15:00:49.970061 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:49.970072 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:49.970113 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:49.970266 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:49.970267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:49.970436 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:49.970499 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:49.970501 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.971174 (+ 673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.971175 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:49.972104 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:49.972107 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.972465 (+ 358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.972467 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:49.972545 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:49.972981 (+ 436us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:49.972988 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:49.974928 (+ 1940us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 15:00:49.975081 17500 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:49.970075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:44025 (local address 127.8.102.62:41841)
0926 15:00:49.970115 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:49.970117 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:49.970126 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:49.970177 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:49.970177 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:49.970187 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:49.970237 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:49.970578 (+ 341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.971095 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.972205 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.972390 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.972440 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:49.975003 (+ 2563us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.2:44025: BlockingRecv error: recv got EOF from 127.8.102.2:44025 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:49 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:49.980216 17631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:49.975237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.2:41853)
0926 15:00:49.975362 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:49.975371 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:49.975413 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:49.975569 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:49.975570 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:49.975711 (+ 141us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:49.975743 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:49.975744 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.976448 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.976449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:49.977263 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:49.977264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.977717 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.977718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:49.977771 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:49.978175 (+ 404us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:49.978180 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:49.980126 (+ 1946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
W20260926 15:00:49.980268 17500 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:49.975296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:41853 (local address 127.8.102.62:41841)
0926 15:00:49.975328 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:49.975330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:49.975420 (+ 90us) server_negotiation.cc:407] Connection header received
0926 15:00:49.975474 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:49.975474 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:49.975491 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:49.975546 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:49.975851 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.976370 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.977382 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:49.977645 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:49.977708 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:49.980201 (+ 2493us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.2:41853: BlockingRecv error: recv got EOF from 127.8.102.2:41853 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
W20260926 15:00:49.980361 17629 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 15:00:50.049999 17632 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:00:50.050278 17632 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:00:50.050304 17632 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:00:50.051740 17632 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:00:50.051783 17632 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:00:50.051812 17632 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:00:50.053217 17632 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:34453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/krb5kdc/oryx_127.8.102.3.keytab
--principal=oryx/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:33963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=36315
--tserver_master_addrs=127.8.102.62:41841
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17632
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:00:50.053412 17632 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:00:50.053596 17632 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:00:50.058986 17641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:00:50.058986 17639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:50.059048 17632 server_base.cc:1034] running on GCE node
W20260926 15:00:50.058988 17638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:00:50.059439 17632 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:00:50.059751 17632 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:00:50.060953 17632 hybrid_clock.cc:648] HybridClock initialized: now 1790434850060918 us; error 45 us; skew 500 ppm
Sep 26 15:00:50 dist-test-slave-jnhs krb5kdc[16763](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434850, etypes {rep=17 tkt=17 ses=17}, oryx/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:00:50.063702 17632 init.cc:377] Logged in from keytab as oryx/127.8.102.3@KRBTEST.COM (short username oryx)
I20260926 15:00:50.064193 17632 webserver.cc:460] Webserver started at http://127.8.102.3:36315/ using document root <none> and password file <none>
I20260926 15:00:50.064432 17632 fs_manager.cc:359] Metadata directory not provided
I20260926 15:00:50.064518 17632 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:00:50.065975 17632 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:50.066619 17648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:50.066834 17632 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:00:50.066944 17632 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "da79b95282a8406899b34479fefb98f3"
format_stamp: "Formatted at 2026-09-26 15:00:49 on dist-test-slave-jnhs"
server_key: "3bc408d5c0474ee2967e70568fb9e99d"
server_key_iv: "aad91932fcf80f080e4ca135aba499ef"
server_key_version: "encryptionkey@0"
I20260926 15:00:50.067312 17632 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:00:50.081107 17632 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:00:50.083477 17632 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:00:50.083654 17632 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:00:50.083912 17632 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:00:50.084228 17632 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:00:50.084282 17632 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:50.084326 17632 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:00:50.084357 17632 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:00:50.090339 17632 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:33963
I20260926 15:00:50.090378 17761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:33963 every 8 connection(s)
I20260926 15:00:50.090762 17632 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
Sep 26 15:00:50 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:50 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:50.098420 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:50.091517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.3:43605)
0926 15:00:50.091720 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:50.091764 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:50.092255 (+ 491us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:50.092585 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:50.092589 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:50.093254 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:50.093454 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:50.093459 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.094238 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.094240 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:50.095086 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:50.095088 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.095489 (+ 401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.095490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:50.095603 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:50.096152 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:50.096161 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:50.098272 (+ 2111us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":93,"threads_started":1}
W20260926 15:00:50.098549 17500 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:50.091539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:43605 (local address 127.8.102.62:41841)
0926 15:00:50.091610 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:50.091612 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:50.091890 (+ 278us) server_negotiation.cc:407] Connection header received
0926 15:00:50.092353 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:50.092354 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:50.092365 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:50.092472 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:50.093578 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.094130 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.095185 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.095404 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.095470 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:50.098435 (+ 2965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.3:43605: BlockingRecv error: recv got EOF from 127.8.102.3:43605 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20260926 15:00:50.098691 17762 heartbeater.cc:644] Failed to heartbeat to 127.8.102.62:41841 (0 consecutive failures): Not authorized: Failed to ping master at 127.8.102.62:41841: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
I20260926 15:00:50.099359 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 17632
Sep 26 15:00:50 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:50 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:50.104014 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:50.098923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.3:54735)
0926 15:00:50.098973 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:50.098984 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:50.099031 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:50.099371 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:50.099372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:50.099578 (+ 206us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:50.099629 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:50.099631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.100507 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.100508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:50.101076 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:50.101078 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.101483 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.101484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:50.101562 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:50.102051 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:50.102057 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:50.103919 (+ 1862us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 15:00:50.104065 17500 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:50.099138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:54735 (local address 127.8.102.62:41841)
0926 15:00:50.099218 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:50.099220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:50.099230 (+ 10us) server_negotiation.cc:407] Connection header received
0926 15:00:50.099251 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:50.099252 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:50.099262 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:50.099366 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:50.099773 (+ 407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.100425 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.101183 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.101388 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.101482 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:50.103989 (+ 2507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.3:54735: BlockingRecv error: recv got EOF from 127.8.102.3:54735 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
I20260926 15:00:50.107582 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:50.099780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49192 (local address 127.8.102.62:41841)
0926 15:00:50.099847 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:50.099849 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:50.099979 (+ 130us) server_negotiation.cc:407] Connection header received
0926 15:00:50.100007 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:50.100008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:50.100021 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:50.100090 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:50.100635 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.101095 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.101950 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.102251 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.102329 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:50.103861 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:00:50.103866 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:00:50.103866 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:00:50.103878 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:00:50.106026 (+ 2148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:50.106411 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:50.106415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:50.106415 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:50.106529 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:00:50.106799 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:00:50.106800 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:00:50.106801 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:00:50.107052 (+ 251us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:00:50.107223 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 15:00:50.107353 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 15:00:50.107485 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
Sep 26 15:00:50 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:50 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:50.108983 17764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:50.104328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.3:60775)
0926 15:00:50.104461 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:50.104470 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:50.104518 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:50.104647 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:50.104647 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:50.104750 (+ 103us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:50.104776 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:50.104777 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.105739 (+ 962us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.105740 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:50.106213 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:50.106214 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.106585 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.106586 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:50.106666 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:50.107046 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:50.107051 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:50.108937 (+ 1886us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":119}
W20260926 15:00:50.109109 17500 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:50.104383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:60775 (local address 127.8.102.62:41841)
0926 15:00:50.104426 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:50.104428 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:50.104525 (+ 97us) server_negotiation.cc:407] Connection header received
0926 15:00:50.104555 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:50.104556 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:50.104567 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:50.104630 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:50.104872 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.105645 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.106318 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.106501 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.106585 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:50.109011 (+ 2426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.3:60775: BlockingRecv error: recv got EOF from 127.8.102.3:60775 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 15:00:50.109119 17762 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 15:00:50 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:50 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:50.854765 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:50.849648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.1:52137)
0926 15:00:50.849794 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:50.849814 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:50.849877 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:50.850132 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:50.850134 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:50.850373 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:50.850423 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:50.850425 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.851273 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.851274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:50.851973 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:50.851975 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.852304 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.852304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:50.852345 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:50.852891 (+ 546us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:50.852896 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:50.854663 (+ 1767us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":91,"threads_started":1}
W20260926 15:00:50.854858 17767 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:50.849750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:52137 (local address 127.8.102.62:41841)
0926 15:00:50.849926 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:50.849931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:50.849941 (+ 10us) server_negotiation.cc:407] Connection header received
0926 15:00:50.849982 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:50.849983 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:50.849993 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:50.850068 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:50.850549 (+ 481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.851185 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.852048 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.852239 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.852293 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:50.854785 (+ 2492us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.1:52137: BlockingRecv error: recv got EOF from 127.8.102.1:52137 (error 108)
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":104,"threads_started":1}
Sep 26 15:00:50 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:50 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:50.986016 17768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:50.980904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.2:51441)
0926 15:00:50.981048 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:50.981068 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:50.981135 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:50.981304 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:50.981305 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:50.981486 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:50.981548 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:50.981550 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.982211 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.982211 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:50.982799 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:50.982801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.983172 (+ 371us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.983173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:50.983224 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:50.983787 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:50.983792 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:50.985934 (+ 2142us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":74,"threads_started":1}
W20260926 15:00:50.986155 17767 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:50.980976 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:51441 (local address 127.8.102.62:41841)
0926 15:00:50.981075 (+ 99us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:50.981076 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:50.981119 (+ 43us) server_negotiation.cc:407] Connection header received
0926 15:00:50.981185 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:50.981185 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:50.981195 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:50.981255 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:50.981626 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.982137 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.982868 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:50.983065 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:50.983120 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:50.986071 (+ 2951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.2:51441: BlockingRecv error: recv got EOF from 127.8.102.2:51441 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
Sep 26 15:00:51 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:51 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:51.114396 17769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:51.109520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.3:51007)
0926 15:00:51.109655 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:51.109666 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:51.109734 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:51.109865 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:51.109866 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:51.110030 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:51.110078 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:51.110080 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:51.110773 (+ 693us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:51.110775 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:51.111343 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:51.111345 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:51.111772 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:51.111774 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:51.111836 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:51.112392 (+ 556us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:51.112398 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:51.114322 (+ 1924us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
W20260926 15:00:51.114593 17767 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:51.109614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:51007 (local address 127.8.102.62:41841)
0926 15:00:51.109659 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:51.109661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:51.109739 (+ 78us) server_negotiation.cc:407] Connection header received
0926 15:00:51.109789 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:51.109790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:51.109802 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:51.109844 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:51.110145 (+ 301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:51.110688 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:51.111456 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:51.111667 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:51.111741 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:51.114509 (+ 2768us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.3:51007: BlockingRecv error: recv got EOF from 127.8.102.3:51007 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
Sep 26 15:00:51 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:51 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:51.860373 17770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:51.855272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.1:39465)
0926 15:00:51.855413 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:51.855432 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:51.855488 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:51.855731 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:51.855731 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:51.855894 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:51.855942 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:51.855944 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:51.856699 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:51.856700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:51.857273 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:51.857274 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:51.857745 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:51.857746 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:51.857803 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:51.858280 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:51.858286 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:51.860259 (+ 1973us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":89,"threads_started":1}
W20260926 15:00:51.860478 17771 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:51.855454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:39465 (local address 127.8.102.62:41841)
0926 15:00:51.855590 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:51.855592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:51.855601 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:51.855634 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:51.855634 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:51.855644 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:51.855710 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:51.856041 (+ 331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:51.856586 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:51.857373 (+ 787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:51.857638 (+ 265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:51.857706 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:51.860404 (+ 2698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.1:39465: BlockingRecv error: recv got EOF from 127.8.102.1:39465 (error 108)
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":94,"threads_started":1}
Sep 26 15:00:51 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:51 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:51.991663 17772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:51.986516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.2:39861)
0926 15:00:51.986695 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:51.986705 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:51.986761 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:51.986931 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:51.986932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:51.987105 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:51.987155 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:51.987157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:51.988017 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:51.988018 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:51.988588 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:51.988590 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:51.988999 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:51.989000 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:51.989056 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:51.989579 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:51.989584 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:51.991592 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":95,"threads_started":1}
W20260926 15:00:51.991833 17771 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:51.986618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:39861 (local address 127.8.102.62:41841)
0926 15:00:51.986698 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:51.986700 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:51.986752 (+ 52us) server_negotiation.cc:407] Connection header received
0926 15:00:51.986809 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:51.986810 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:51.986821 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:51.986882 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:51.987267 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:51.987917 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:51.988693 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:51.988916 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:51.988986 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:51.991725 (+ 2739us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.2:39861: BlockingRecv error: recv got EOF from 127.8.102.2:39861 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Sep 26 15:00:52 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:52 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:52.120563 17773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:52.115041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.3:44615)
0926 15:00:52.115226 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:52.115240 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:52.115323 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:52.115505 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:52.115506 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:52.115683 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:52.115731 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:52.115733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:52.116569 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:52.116570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:52.117178 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:52.117179 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:52.117764 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:52.117765 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:52.117841 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:52.118415 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:52.118421 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:52.120492 (+ 2071us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":162,"thread_start_us":92,"threads_started":1}
W20260926 15:00:52.120735 17771 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:52.115217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:44615 (local address 127.8.102.62:41841)
0926 15:00:52.115292 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:52.115294 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:52.115308 (+ 14us) server_negotiation.cc:407] Connection header received
0926 15:00:52.115393 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:52.115394 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:52.115406 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:52.115465 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:52.115830 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:52.116449 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:52.117259 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:52.117676 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:52.117766 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:52.120652 (+ 2886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.3:44615: BlockingRecv error: recv got EOF from 127.8.102.3:44615 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Sep 26 15:00:52 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:52 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:52.869922 17774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:52.860890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.1:60933)
0926 15:00:52.861018 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:52.861030 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:52.861098 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:52.861322 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:52.861323 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:52.861484 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:52.861534 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:52.861536 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:52.862264 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:52.862265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:52.862948 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:52.862951 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:52.863385 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:52.863386 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:52.863433 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:52.867574 (+ 4141us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:52.867580 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:52.869849 (+ 2269us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
W20260926 15:00:52.870085 17775 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:52.861025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:60933 (local address 127.8.102.62:41841)
0926 15:00:52.861189 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:52.861190 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:52.861198 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:00:52.861219 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:52.861220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:52.861229 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:52.861294 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:52.861642 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:52.862165 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:52.863046 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:52.863261 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:52.863376 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:52.869974 (+ 6598us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.1:60933: BlockingRecv error: recv got EOF from 127.8.102.1:60933 (error 108)
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":105,"threads_started":1}
Sep 26 15:00:52 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:52 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:52.997735 17776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:52.992278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.2:44101)
0926 15:00:52.992490 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:52.992503 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:52.992583 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:52.992853 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:52.992855 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:52.993135 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:52.993194 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:52.993197 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:52.994000 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:52.994001 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:52.994562 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:52.994564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:52.995133 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:52.995135 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:52.995218 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:52.995789 (+ 571us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:52.995795 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:52.997663 (+ 1868us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":100,"threads_started":1}
W20260926 15:00:52.997876 17775 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:52.992407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:44101 (local address 127.8.102.62:41841)
0926 15:00:52.992484 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:52.992487 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:52.992562 (+ 75us) server_negotiation.cc:407] Connection header received
0926 15:00:52.992711 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:52.992712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:52.992726 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:52.992800 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:52.993328 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:52.993910 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:52.994690 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:52.994977 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:52.995047 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:52.997788 (+ 2741us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.2:44101: BlockingRecv error: recv got EOF from 127.8.102.2:44101 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 26 15:00:53 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:53 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:53.125974 17777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:53.121051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.3:60347)
0926 15:00:53.121223 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:53.121235 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:53.121307 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:53.121461 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:53.121463 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:53.121635 (+ 172us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:53.121686 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:53.121688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:53.122520 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:53.122521 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:53.123188 (+ 667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:53.123191 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:53.123560 (+ 369us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:53.123561 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:53.123601 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:53.124067 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:53.124071 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:53.125902 (+ 1831us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":101,"threads_started":1}
W20260926 15:00:53.126139 17775 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:53.121176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:60347 (local address 127.8.102.62:41841)
0926 15:00:53.121234 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:53.121235 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:53.121308 (+ 73us) server_negotiation.cc:407] Connection header received
0926 15:00:53.121355 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:53.121355 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:53.121366 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:53.121416 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:53.121795 (+ 379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:53.122417 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:53.123283 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:53.123487 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:53.123552 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:53.126052 (+ 2500us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.3:60347: BlockingRecv error: recv got EOF from 127.8.102.3:60347 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 15:00:53 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:53 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:53.875305 17778 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:53.870481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.1:60337)
0926 15:00:53.870623 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:53.870662 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:53.870741 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:53.870911 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:53.870912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:53.871079 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:53.871174 (+ 95us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:53.871176 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:53.871929 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:53.871930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:53.872469 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:53.872470 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:53.872831 (+ 361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:53.872832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:53.872881 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:53.873446 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:53.873452 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:53.875246 (+ 1794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":86,"threads_started":1}
W20260926 15:00:53.875449 17779 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:53.870531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:60337 (local address 127.8.102.62:41841)
0926 15:00:53.870679 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:53.870681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:53.870726 (+ 45us) server_negotiation.cc:407] Connection header received
0926 15:00:53.870800 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:53.870800 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:53.870810 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:53.870863 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:53.871342 (+ 479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:53.871847 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:53.872551 (+ 704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:53.872743 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:53.872800 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:53.875335 (+ 2535us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.1:60337: BlockingRecv error: recv got EOF from 127.8.102.1:60337 (error 108)
Metrics: {"server-negotiator.queue_time_us":125,"thread_start_us":65,"threads_started":1}
Sep 26 15:00:53 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:53 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:54.003065 17780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:53.998266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.2:44365)
0926 15:00:53.998406 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:53.998421 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:53.998473 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:53.998720 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:53.998720 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:53.998878 (+ 158us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:53.998928 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:53.998930 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:53.999604 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:53.999604 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:54.000129 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:54.000131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:54.000644 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:54.000645 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:54.000710 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:54.001214 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:54.001219 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:54.003004 (+ 1785us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":82,"threads_started":1}
W20260926 15:00:54.003206 17779 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:53.998339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:44365 (local address 127.8.102.62:41841)
0926 15:00:53.998434 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:53.998463 (+ 29us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:53.998475 (+ 12us) server_negotiation.cc:407] Connection header received
0926 15:00:53.998571 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:53.998572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:53.998584 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:53.998713 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:53.999004 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:53.999527 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:54.000244 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:54.000562 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:54.000639 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:54.003118 (+ 2479us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.2:44365: BlockingRecv error: recv got EOF from 127.8.102.2:44365 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 26 15:00:54 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:54 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:54.131354 17781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:54.126534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.3:39529)
0926 15:00:54.126710 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:54.126720 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:54.126781 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:54.126925 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:54.126926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:54.127081 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:54.127128 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:54.127130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:54.127885 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:54.127887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:54.128498 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:54.128501 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:54.128852 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:54.128853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:54.128891 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:54.129475 (+ 584us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:54.129480 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:54.131295 (+ 1815us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":158,"thread_start_us":84,"threads_started":1}
W20260926 15:00:54.131554 17779 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:54.126686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:39529 (local address 127.8.102.62:41841)
0926 15:00:54.126743 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:54.126745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:54.126764 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:00:54.126836 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:54.126837 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:54.126851 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:54.126917 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:54.127222 (+ 305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:54.127771 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:54.128593 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:54.128784 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:54.128844 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:54.131426 (+ 2582us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.3:39529: BlockingRecv error: recv got EOF from 127.8.102.3:39529 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 26 15:00:54 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:54 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:54.881007 17782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:54.875820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.1:38637)
0926 15:00:54.875986 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:54.875997 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:54.876062 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:54.876219 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:54.876220 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:54.876396 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:54.876447 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:54.876448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:54.877165 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:54.877167 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:54.878063 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:54.878065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:54.878477 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:54.878478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:54.878526 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:54.879108 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:54.879114 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:54.880919 (+ 1805us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":103,"threads_started":1}
W20260926 15:00:54.881282 17783 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:54.875940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:38637 (local address 127.8.102.62:41841)
0926 15:00:54.876091 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:54.876093 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:54.876103 (+ 10us) server_negotiation.cc:407] Connection header received
0926 15:00:54.876129 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:54.876130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:54.876140 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:54.876202 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:54.876539 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:54.877036 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:54.878152 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:54.878359 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:54.878421 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:54.881174 (+ 2753us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.1:38637: BlockingRecv error: recv got EOF from 127.8.102.1:38637 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":95,"threads_started":1}
Sep 26 15:00:55 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:55 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:55.008674 17784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:55.003585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.2:37621)
0926 15:00:55.003751 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:55.003764 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:55.003863 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:55.004022 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:55.004023 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:55.004211 (+ 188us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:55.004260 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:55.004262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:55.004959 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:55.004960 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:55.005489 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:55.005491 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:55.005903 (+ 412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:55.005904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:55.005963 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:55.006436 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:55.006440 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:55.008587 (+ 2147us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":93,"threads_started":1}
W20260926 15:00:55.008833 17783 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:55.003707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:37621 (local address 127.8.102.62:41841)
0926 15:00:55.003762 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:55.003765 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:55.003847 (+ 82us) server_negotiation.cc:407] Connection header received
0926 15:00:55.003951 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:55.003952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:55.003965 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:55.004008 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:55.004351 (+ 343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:55.004864 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:55.005596 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:55.005809 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:55.005900 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:55.008747 (+ 2847us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.2:37621: BlockingRecv error: recv got EOF from 127.8.102.2:37621 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
Sep 26 15:00:55 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:55 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:55.136461 17785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:55.131808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.3:48849)
0926 15:00:55.131943 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:55.131973 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:55.132036 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:55.132197 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:55.132198 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:55.132374 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:55.132418 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:55.132420 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:55.133092 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:55.133094 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:55.133621 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:55.133623 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:55.133977 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:55.133978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:55.134041 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:55.134511 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:55.134515 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:55.136400 (+ 1885us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":74,"threads_started":1}
W20260926 15:00:55.136646 17783 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:55.131934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:48849 (local address 127.8.102.62:41841)
0926 15:00:55.132001 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:55.132003 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:55.132017 (+ 14us) server_negotiation.cc:407] Connection header received
0926 15:00:55.132102 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:55.132103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:55.132118 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:55.132191 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:55.132499 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:55.132994 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:55.133707 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:55.133900 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:55.133979 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:55.136570 (+ 2591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.3:48849: BlockingRecv error: recv got EOF from 127.8.102.3:48849 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 15:00:55 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:55 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:55.886703 17786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:55.881515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.1:58227)
0926 15:00:55.881654 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:55.881666 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:55.881718 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:55.881911 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:55.881912 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:55.882079 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:55.882132 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:55.882133 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:55.882859 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:55.882860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:55.883410 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:55.883411 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:55.883904 (+ 493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:55.883905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:55.883959 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:55.884522 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:55.884527 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:55.886617 (+ 2090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":80,"threads_started":1}
W20260926 15:00:55.886898 17787 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:55.881643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:58227 (local address 127.8.102.62:41841)
0926 15:00:55.881779 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:55.881781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:55.881789 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:00:55.881814 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:55.881815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:55.881825 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:55.881892 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:55.882206 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:55.882766 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:55.883541 (+ 775us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:55.883778 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:55.883854 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:55.886813 (+ 2959us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.1:58227: BlockingRecv error: recv got EOF from 127.8.102.1:58227 (error 108)
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":70,"threads_started":1}
Sep 26 15:00:56 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:56 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:56.014621 17788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:56.009179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.2:38991)
0926 15:00:56.009345 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:56.009376 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:56.009443 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:56.009660 (+ 217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:56.009661 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:56.009843 (+ 182us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:56.009891 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:56.009893 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:56.010659 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:56.010661 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:56.011404 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:56.011407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:56.012010 (+ 603us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:56.012012 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:56.012083 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:56.012664 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:56.012670 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:56.014558 (+ 1888us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":81,"threads_started":1}
W20260926 15:00:56.014843 17787 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:56.009338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:38991 (local address 127.8.102.62:41841)
0926 15:00:56.009412 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:56.009414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:56.009427 (+ 13us) server_negotiation.cc:407] Connection header received
0926 15:00:56.009571 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:56.009572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:56.009583 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:56.009654 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:56.009979 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:56.010545 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:56.011522 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:56.011858 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:56.011944 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:56.014767 (+ 2823us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.2:38991: BlockingRecv error: recv got EOF from 127.8.102.2:38991 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 26 15:00:56 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:56 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:56.142390 17789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:56.137016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.3:49977)
0926 15:00:56.137171 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:56.137190 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:56.137266 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:56.137443 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:56.137444 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:56.137653 (+ 209us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:56.137700 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:56.137702 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:56.138586 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:56.138588 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:56.139250 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:56.139252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:56.139645 (+ 393us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:56.139646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:56.139703 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:56.140215 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:56.140220 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:56.142282 (+ 2062us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":91,"threads_started":1}
W20260926 15:00:56.142539 17787 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:56.137153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:49977 (local address 127.8.102.62:41841)
0926 15:00:56.137240 (+ 87us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:56.137243 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:56.137252 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:00:56.137359 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:56.137360 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:56.137378 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:56.137436 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:56.137927 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:56.138501 (+ 574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:56.139344 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:56.139556 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:56.139644 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:56.142436 (+ 2792us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.3:49977: BlockingRecv error: recv got EOF from 127.8.102.3:49977 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Sep 26 15:00:56 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:56 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:56.892210 17790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:56.887229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.1:48469)
0926 15:00:56.887462 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:56.887476 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:56.887548 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:56.887694 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:56.887695 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:56.887859 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:56.887912 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:56.887913 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:56.888663 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:56.888664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:56.889215 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:56.889217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:56.889657 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:56.889658 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:56.889705 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:56.890181 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:56.890185 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:56.892147 (+ 1962us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":210,"thread_start_us":69,"threads_started":1}
W20260926 15:00:56.892392 17791 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:56.887365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:48469 (local address 127.8.102.62:41841)
0926 15:00:56.887500 (+ 135us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:56.887502 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:56.887531 (+ 29us) server_negotiation.cc:407] Connection header received
0926 15:00:56.887604 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:56.887605 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:56.887615 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:56.887664 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:56.888010 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:56.888556 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:56.889329 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:56.889573 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:56.889642 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:56.892281 (+ 2639us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.1:48469: BlockingRecv error: recv got EOF from 127.8.102.1:48469 (error 108)
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":64,"threads_started":1}
Sep 26 15:00:57 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:57 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:57.020753 17792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:57.015237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.2:41917)
0926 15:00:57.015469 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:57.015480 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:57.015554 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:57.015760 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:57.015761 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:57.015963 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:57.016016 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:57.016018 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:57.016755 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:57.016756 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:57.017364 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:57.017366 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:57.017820 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:57.017821 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:57.017873 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:57.018364 (+ 491us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:57.018369 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:57.020676 (+ 2307us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":103,"threads_started":1}
W20260926 15:00:57.021013 17791 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:57.015346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:41917 (local address 127.8.102.62:41841)
0926 15:00:57.015419 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:57.015421 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:57.015571 (+ 150us) server_negotiation.cc:407] Connection header received
0926 15:00:57.015609 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:57.015610 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:57.015623 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:57.015692 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:57.016110 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:57.016659 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:57.017507 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:57.017716 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:57.017794 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:57.020946 (+ 3152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.2:41917: BlockingRecv error: recv got EOF from 127.8.102.2:41917 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 15:00:57 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:57 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:57.147933 17793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:57.142972 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.3:40199)
0926 15:00:57.143182 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:57.143195 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:57.143289 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:57.143515 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:57.143516 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:57.143702 (+ 186us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:57.143748 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:57.143750 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:57.144430 (+ 680us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:57.144431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:57.144981 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:57.144983 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:57.145374 (+ 391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:57.145375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:57.145434 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:57.145949 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:57.145954 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:57.147863 (+ 1909us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":192,"thread_start_us":100,"threads_started":1}
W20260926 15:00:57.148078 17791 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:57.143077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:40199 (local address 127.8.102.62:41841)
0926 15:00:57.143167 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:57.143170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:57.143268 (+ 98us) server_negotiation.cc:407] Connection header received
0926 15:00:57.143422 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:57.143423 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:57.143435 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:57.143506 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:57.143838 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:57.144334 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:57.145073 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:57.145275 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:57.145374 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:57.147991 (+ 2617us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.3:40199: BlockingRecv error: recv got EOF from 127.8.102.3:40199 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Sep 26 15:00:57 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:57 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:57.904687 17794 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:57.892860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.1:45557)
0926 15:00:57.893063 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:57.893079 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:57.893154 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:57.893327 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:57.893383 (+ 56us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:57.893617 (+ 234us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:57.893698 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:57.893702 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:57.894559 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:57.894561 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:57.895169 (+ 608us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:57.895171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:57.895624 (+ 453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:57.895625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:57.895690 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:57.902366 (+ 6676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:57.902374 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:57.904583 (+ 2209us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":62,"threads_started":1}
W20260926 15:00:57.904775 17795 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:57.893019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:45557 (local address 127.8.102.62:41841)
0926 15:00:57.893171 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:57.893173 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:57.893187 (+ 14us) server_negotiation.cc:407] Connection header received
0926 15:00:57.893219 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:57.893220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:57.893235 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:57.893304 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:57.893773 (+ 469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:57.894458 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:57.895268 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:57.895511 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:57.895621 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:57.904690 (+ 9069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.1:45557: BlockingRecv error: recv got EOF from 127.8.102.1:45557 (error 108)
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":93,"threads_started":1}
Sep 26 15:00:58 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:58 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:58.026346 17796 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:58.021359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.2:36387)
0926 15:00:58.021513 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:58.021545 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:58.021614 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:58.021798 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:58.021799 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:58.021983 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:58.022031 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:58.022033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:58.022966 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:58.022967 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:58.023500 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:58.023502 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:58.023880 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:58.023881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:58.023940 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:58.024465 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:58.024470 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:58.026249 (+ 1779us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":92,"threads_started":1}
W20260926 15:00:58.026396 17795 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:58.021501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:36387 (local address 127.8.102.62:41841)
0926 15:00:58.021576 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:58.021578 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:58.021595 (+ 17us) server_negotiation.cc:407] Connection header received
0926 15:00:58.021700 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:58.021701 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:58.021721 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:58.021789 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:58.022122 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:58.022868 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:58.023591 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:58.023789 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:58.023878 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:58.026330 (+ 2452us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.2:36387: BlockingRecv error: recv got EOF from 127.8.102.2:36387 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 15:00:58 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:58 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:58.153582 17797 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:58.148427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.3:49663)
0926 15:00:58.148563 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:58.148575 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:58.148644 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:58.148780 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:58.148781 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:58.148938 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:58.148985 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:58.148987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:58.149830 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:58.149831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:58.150372 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:58.150373 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:58.150998 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:58.151000 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:58.151054 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:58.151569 (+ 515us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:58.151573 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:58.153480 (+ 1907us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":116,"threads_started":1}
W20260926 15:00:58.153637 17795 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:58.148485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:49663 (local address 127.8.102.62:41841)
0926 15:00:58.148569 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:58.148571 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:58.148648 (+ 77us) server_negotiation.cc:407] Connection header received
0926 15:00:58.148705 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:58.148705 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:58.148717 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:58.148760 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:58.149092 (+ 332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:58.149719 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:58.150638 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:58.150907 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:58.150974 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:58.153570 (+ 2596us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.3:49663: BlockingRecv error: recv got EOF from 127.8.102.3:49663 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Sep 26 15:00:58 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:58 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:58.910591 17798 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:58.905264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.1:57049)
0926 15:00:58.905429 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:58.905448 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:58.905509 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:58.905751 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:58.905751 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:58.905914 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:58.905971 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:58.905972 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:58.906783 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:58.906784 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:58.907319 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:58.907321 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:58.907724 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:58.907725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:58.907797 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:58.908308 (+ 511us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:58.908312 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:58.910483 (+ 2171us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":104,"threads_started":1}
W20260926 15:00:58.910743 17799 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:58.905449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:57049 (local address 127.8.102.62:41841)
0926 15:00:58.905593 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:58.905596 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:58.905606 (+ 10us) server_negotiation.cc:407] Connection header received
0926 15:00:58.905643 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:58.905644 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:58.905655 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:58.905730 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:58.906072 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:58.906658 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:58.907438 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:58.907650 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:58.907717 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:58.910614 (+ 2897us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.1:57049: BlockingRecv error: recv got EOF from 127.8.102.1:57049 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":79,"threads_started":1}
Sep 26 15:00:59 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:59 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:59.032455 17800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:59.026967 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.2:52773)
0926 15:00:59.027126 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:59.027138 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:59.027260 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:59.027454 (+ 194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:59.027454 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:59.027693 (+ 239us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:59.027761 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:59.027763 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:59.028720 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:59.028721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:59.029267 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:59.029269 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:59.029685 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:59.029686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:59.029745 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:59.030284 (+ 539us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:59.030288 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:59.032357 (+ 2069us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":64,"threads_started":1}
W20260926 15:00:59.032528 17799 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:59.027080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:52773 (local address 127.8.102.62:41841)
0926 15:00:59.027149 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:59.027151 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:59.027252 (+ 101us) server_negotiation.cc:407] Connection header received
0926 15:00:59.027358 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:59.027359 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:59.027371 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:59.027435 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:59.027865 (+ 430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:59.028620 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:59.029362 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:59.029586 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:59.029683 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:59.032452 (+ 2769us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.2:52773: BlockingRecv error: recv got EOF from 127.8.102.2:52773 (error 108)
Metrics: {"server-negotiator.queue_time_us":37}
Sep 26 15:00:59 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:59 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:59.159039 17801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:59.154057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.3:60091)
0926 15:00:59.154212 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:59.154232 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:59.154290 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:59.154440 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:59.154441 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:59.154606 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:59.154710 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:59.154712 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:59.155434 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:59.155435 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:59.156004 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:59.156006 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:59.156384 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:59.156386 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:59.156428 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:59.156911 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:59.156915 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:59.158965 (+ 2050us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":96,"threads_started":1}
W20260926 15:00:59.159143 17799 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:59.154132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:60091 (local address 127.8.102.62:41841)
0926 15:00:59.154206 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:59.154208 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:59.154294 (+ 86us) server_negotiation.cc:407] Connection header received
0926 15:00:59.154336 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:59.154337 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:59.154349 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:59.154404 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:59.154790 (+ 386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:59.155342 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:59.156087 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:59.156312 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:59.156370 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:59.159068 (+ 2698us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.3:60091: BlockingRecv error: recv got EOF from 127.8.102.3:60091 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
Sep 26 15:00:59 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:00:59 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:00:59.916425 17802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:00:59.911168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.1:37767)
0926 15:00:59.911338 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 15:00:59.911351 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:00:59.911464 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:00:59.911633 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:00:59.911634 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:00:59.911827 (+ 193us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:00:59.911874 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:59.911876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:59.912581 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:59.912582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:59.913123 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:00:59.913124 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:00:59.913524 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:00:59.913525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:00:59.913592 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:59.914121 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:00:59.914126 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:00:59.916279 (+ 2153us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":71,"threads_started":1}
W20260926 15:00:59.916553 17803 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:00:59.911232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:37767 (local address 127.8.102.62:41841)
0926 15:00:59.911347 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0926 15:00:59.911350 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:00:59.911491 (+ 141us) server_negotiation.cc:407] Connection header received
0926 15:00:59.911524 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:00:59.911525 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:00:59.911540 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:00:59.911626 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:00:59.911968 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:59.912484 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:59.913217 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:00:59.913425 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:00:59.913524 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:00:59.916456 (+ 2932us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.1:37767: BlockingRecv error: recv got EOF from 127.8.102.1:37767 (error 108)
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":43,"threads_started":1}
Sep 26 15:01:00 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:01:00 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:01:00.039153 17804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:00.033131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.2:36023)
0926 15:01:00.033422 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:00.033436 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:00.033546 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:00.033736 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:00.033737 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:00.034050 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:00.034105 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:00.034108 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:00.034994 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:00.034995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:00.035613 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:00.035615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:00.036031 (+ 416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:00.036032 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:00.036095 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:00.036597 (+ 502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:00.036601 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:00.039062 (+ 2461us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":245,"thread_start_us":116,"threads_started":1}
W20260926 15:01:00.039333 17803 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:01:00.033254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:36023 (local address 127.8.102.62:41841)
0926 15:01:00.033421 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:00.033424 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:00.033524 (+ 100us) server_negotiation.cc:407] Connection header received
0926 15:01:00.033625 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:00.033626 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:00.033639 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:00.033713 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:00.034241 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:00.034896 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:00.035727 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:00.035949 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:00.036034 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:00.039244 (+ 3210us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.2:36023: BlockingRecv error: recv got EOF from 127.8.102.2:36023 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
Sep 26 15:01:00 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:01:00 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:01:00.164532 17805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:00.159619 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.3:50291)
0926 15:01:00.159750 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:00.159763 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:00.159824 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:00.160057 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:00.160058 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:00.160222 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:00.160269 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:00.160270 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:00.160942 (+ 672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:00.160943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:00.161477 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:00.161478 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:00.161798 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:00.161799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:00.161838 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:00.162303 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:00.162308 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:00.164426 (+ 2118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":81,"threads_started":1}
W20260926 15:01:00.164590 17803 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:01:00.159829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:50291 (local address 127.8.102.62:41841)
0926 15:01:00.159917 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:00.159923 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:00.159932 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:01:00.159959 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:00.159960 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:00.159972 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:00.160033 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:00.160341 (+ 308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:00.160840 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:00.161553 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:00.161732 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:00.161786 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:00.164521 (+ 2735us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.3:50291: BlockingRecv error: recv got EOF from 127.8.102.3:50291 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Sep 26 15:01:00 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:01:00 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:01:00.922780 17806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:00.917043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.1:49113)
0926 15:01:00.917212 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:00.917223 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:00.917276 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:00.917444 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:00.917445 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:00.917687 (+ 242us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:00.917735 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:00.917737 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:00.918587 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:00.918588 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:00.919391 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:00.919394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:00.919850 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:00.919851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:00.919903 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:00.920386 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:00.920391 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:00.922696 (+ 2305us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":103,"threads_started":1}
W20260926 15:01:00.922874 17807 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:01:00.917137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:49113 (local address 127.8.102.62:41841)
0926 15:01:00.917280 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:00.917282 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:00.917323 (+ 41us) server_negotiation.cc:407] Connection header received
0926 15:01:00.917350 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:00.917350 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:00.917361 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:00.917421 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:00.917817 (+ 396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:00.918464 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:00.919501 (+ 1037us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:00.919705 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:00.919781 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:00.922801 (+ 3020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.1:49113: BlockingRecv error: recv got EOF from 127.8.102.1:49113 (error 108)
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":93,"threads_started":1}
Sep 26 15:01:01 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:01:01 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:01:01.045209 17808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:01.039703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.2:57925)
0926 15:01:01.039891 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:01.039901 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:01.039955 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:01.040117 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:01.040118 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:01.040288 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:01.040339 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:01.040341 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:01.041090 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:01.041091 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:01.041738 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:01.041741 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:01.042114 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:01.042115 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:01.042163 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:01.042878 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:01.042884 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:01.045137 (+ 2253us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":108,"threads_started":1}
W20260926 15:01:01.045410 17807 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:01:01.039806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:57925 (local address 127.8.102.62:41841)
0926 15:01:01.039932 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:01.039935 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:01.039956 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:01:01.040038 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:01.040039 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:01.040050 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:01.040100 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:01.040446 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:01.041003 (+ 557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:01.041839 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:01.042036 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:01.042097 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:01.045278 (+ 3181us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.2:57925: BlockingRecv error: recv got EOF from 127.8.102.2:57925 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
Sep 26 15:01:01 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:01:01 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:01:01.170081 17809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:01.165169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.3:49075)
0926 15:01:01.165322 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:01.165335 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:01.165401 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:01.165550 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:01.165551 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:01.165741 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:01.165785 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:01.165787 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:01.166492 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:01.166493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:01.167068 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:01.167070 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:01.167452 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:01.167453 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:01.167525 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:01.168002 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:01.168006 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:01.170007 (+ 2001us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":89,"threads_started":1}
W20260926 15:01:01.170212 17807 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:01:01.165255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:49075 (local address 127.8.102.62:41841)
0926 15:01:01.165322 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:01.165324 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:01.165382 (+ 58us) server_negotiation.cc:407] Connection header received
0926 15:01:01.165468 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:01.165469 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:01.165484 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:01.165533 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:01.165872 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:01.166396 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:01.167159 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:01.167358 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:01.167446 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:01.170108 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.3:49075: BlockingRecv error: recv got EOF from 127.8.102.3:49075 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 26 15:01:01 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:01:01 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:01:01.928376 17810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:01.923321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.1:57125)
0926 15:01:01.923520 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:01.923532 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:01.923591 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:01.923737 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:01.923738 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:01.923918 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:01.923980 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:01.923981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:01.924656 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:01.924656 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:01.925222 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:01.925223 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:01.925605 (+ 382us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:01.925606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:01.925647 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:01.926223 (+ 576us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:01.926228 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:01.928298 (+ 2070us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":178,"thread_start_us":108,"threads_started":1}
W20260926 15:01:01.928519 17811 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:01:01.923451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:57125 (local address 127.8.102.62:41841)
0926 15:01:01.923589 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:01.923590 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:01.923600 (+ 10us) server_negotiation.cc:407] Connection header received
0926 15:01:01.923642 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:01.923643 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:01.923654 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:01.923728 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:01.924061 (+ 333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:01.924579 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:01.925308 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:01.925535 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:01.925593 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:01.928419 (+ 2826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.1:57125: BlockingRecv error: recv got EOF from 127.8.102.1:57125 (error 108)
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":78,"threads_started":1}
Sep 26 15:01:02 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:01:02 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:01:02.051258 17812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:02.045810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.2:45223)
0926 15:01:02.045995 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:02.046009 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:02.046095 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:02.046242 (+ 147us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:02.046243 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:02.046416 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:02.046466 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:02.046468 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:02.047153 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:02.047154 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:02.047743 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:02.047745 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:02.048241 (+ 496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:02.048242 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:02.048304 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:02.048963 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:02.048969 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:02.051119 (+ 2150us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":72,"threads_started":1}
W20260926 15:01:02.051298 17811 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:01:02.045969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:45223 (local address 127.8.102.62:41841)
0926 15:01:02.046043 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:02.046044 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:02.046076 (+ 32us) server_negotiation.cc:407] Connection header received
0926 15:01:02.046160 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:02.046161 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:02.046172 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:02.046216 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:02.046538 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:02.047056 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:02.047868 (+ 812us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:02.048088 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:02.048161 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:02.051230 (+ 3069us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.2:45223: BlockingRecv error: recv got EOF from 127.8.102.2:45223 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 15:01:02 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:01:02 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:01:02.175625 17813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:02.170499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.3:35003)
0926 15:01:02.170728 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:02.170747 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:02.170847 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:02.171049 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:02.171050 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:02.171268 (+ 218us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:02.171315 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:02.171317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:02.172021 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:02.172022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:02.172633 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:02.172635 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:02.173044 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:02.173045 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:02.173122 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:02.173646 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:02.173651 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:02.175515 (+ 1864us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":211,"thread_start_us":99,"threads_started":1}
W20260926 15:01:02.175663 17811 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:01:02.170667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:35003 (local address 127.8.102.62:41841)
0926 15:01:02.170731 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:02.170734 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:02.170839 (+ 105us) server_negotiation.cc:407] Connection header received
0926 15:01:02.170944 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:02.170945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:02.170960 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:02.171041 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:02.171410 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:02.171928 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:02.172730 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:02.172947 (+ 217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:02.173036 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:02.175597 (+ 2561us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.3:35003: BlockingRecv error: recv got EOF from 127.8.102.3:35003 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Sep 26 15:01:02 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:01:02 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:01:02.942556 17814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:02.928978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.1:59499)
0926 15:01:02.929114 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:02.929129 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:02.929212 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:02.929413 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:02.929415 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:02.929643 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:02.929701 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:02.929703 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:02.930466 (+ 763us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:02.930467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:02.931073 (+ 606us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:02.931075 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:02.931448 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:02.931448 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:02.931496 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:02.940444 (+ 8948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:02.940465 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:02.942488 (+ 2023us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":77,"threads_started":1}
W20260926 15:01:02.942749 17815 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:01:02.929089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:59499 (local address 127.8.102.62:41841)
0926 15:01:02.929226 (+ 137us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:02.929228 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:02.929238 (+ 10us) server_negotiation.cc:407] Connection header received
0926 15:01:02.929279 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:02.929280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:02.929292 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:02.929361 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:02.929844 (+ 483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:02.930359 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:02.931174 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:02.931371 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:02.931469 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:02.942620 (+ 11151us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.1:59499: BlockingRecv error: recv got EOF from 127.8.102.1:59499 (error 108)
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":66,"threads_started":1}
Sep 26 15:01:03 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:01:03 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:01:03.056644 17816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:03.051867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.2:54541)
0926 15:01:03.052037 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:03.052049 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:03.052121 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:03.052255 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:03.052256 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:03.052425 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:03.052474 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:03.052475 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:03.053138 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:03.053139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:03.053709 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:03.053711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:03.054042 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:03.054043 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:03.054083 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:03.054504 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:03.054509 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:03.056555 (+ 2046us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":84,"threads_started":1}
W20260926 15:01:03.056762 17815 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:01:03.051967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:54541 (local address 127.8.102.62:41841)
0926 15:01:03.052015 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:03.052017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:03.052130 (+ 113us) server_negotiation.cc:407] Connection header received
0926 15:01:03.052194 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:03.052195 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:03.052205 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:03.052251 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:03.052547 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:03.053045 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:03.053790 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:03.053970 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:03.054029 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:03.056653 (+ 2624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.2:54541: BlockingRecv error: recv got EOF from 127.8.102.2:54541 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
Sep 26 15:01:03 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:01:03 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:01:03.180876 17817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:03.176155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.3:46517)
0926 15:01:03.176314 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:03.176326 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:03.176398 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:03.176560 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:03.176561 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:03.176748 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:03.176791 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:03.176792 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:03.177461 (+ 669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:03.177462 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:03.178035 (+ 573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:03.178037 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:03.178404 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:03.178405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:03.178463 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:03.178973 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:03.178979 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:03.180760 (+ 1781us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":95,"threads_started":1}
W20260926 15:01:03.180909 17815 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:01:03.176358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:46517 (local address 127.8.102.62:41841)
0926 15:01:03.176428 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:03.176430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:03.176439 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:01:03.176466 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:03.176469 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:03.176487 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:03.176554 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:03.176873 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:03.177373 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:03.178129 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:03.178320 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:03.178405 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:03.180839 (+ 2434us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.3:46517: BlockingRecv error: recv got EOF from 127.8.102.3:46517 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
Sep 26 15:01:03 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:01:03 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:01:03.948099 17818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:03.943099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.1:37737)
0926 15:01:03.943307 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:03.943318 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:03.943378 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:03.943531 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:03.943532 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:03.943699 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:03.943748 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:03.943753 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:03.944444 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:03.944445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:03.944974 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:03.944976 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:03.945346 (+ 370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:03.945347 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:03.945410 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:03.945939 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:03.945944 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:03.948022 (+ 2078us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":126,"threads_started":1}
W20260926 15:01:03.948251 17819 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:01:03.943229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:37737 (local address 127.8.102.62:41841)
0926 15:01:03.943350 (+ 121us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:03.943352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:03.943362 (+ 10us) server_negotiation.cc:407] Connection header received
0926 15:01:03.943446 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:03.943446 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:03.943458 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:03.943512 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:03.943822 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:03.944368 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:03.945068 (+ 700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:03.945273 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:03.945333 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:03.948155 (+ 2822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.1:37737: BlockingRecv error: recv got EOF from 127.8.102.1:37737 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":56,"threads_started":1}
Sep 26 15:01:04 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:01:04 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:01:04.061759 17820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:04.057158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.2:59191)
0926 15:01:04.057292 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:04.057304 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:04.057363 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:04.057504 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:04.057505 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:04.057689 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:04.057737 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:04.057739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:04.058391 (+ 652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:04.058392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:04.058962 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:04.058964 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:04.059324 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:04.059325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:04.059388 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:04.059853 (+ 465us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:04.059858 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:04.061660 (+ 1802us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":86,"threads_started":1}
W20260926 15:01:04.061817 17819 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:01:04.057219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:59191 (local address 127.8.102.62:41841)
0926 15:01:04.057387 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:04.057389 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:04.057398 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:01:04.057424 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:04.057424 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:04.057435 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:04.057490 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:04.057803 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:04.058311 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:04.059044 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:04.059235 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:04.059293 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:04.061746 (+ 2453us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.2:59191: BlockingRecv error: recv got EOF from 127.8.102.2:59191 (error 108)
Metrics: {"server-negotiator.queue_time_us":135}
Sep 26 15:01:04 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:01:04 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.3@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:01:04.186714 17821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:04.181507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.3:49299)
0926 15:01:04.181643 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:04.181656 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:04.181718 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:04.181872 (+ 154us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:04.181873 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:04.182036 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:04.182082 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:04.182084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:04.182742 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:04.182744 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:04.183378 (+ 634us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:04.183381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:04.183830 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:04.183831 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:04.183892 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:04.184442 (+ 550us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:04.184447 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:04.186580 (+ 2133us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
W20260926 15:01:04.186784 17819 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:01:04.181648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:49299 (local address 127.8.102.62:41841)
0926 15:01:04.181720 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:04.181721 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:04.181761 (+ 40us) server_negotiation.cc:407] Connection header received
0926 15:01:04.181788 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:04.181789 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:04.181800 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:04.181842 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:04.182151 (+ 309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:04.182676 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:04.183525 (+ 849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:04.183727 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:04.183810 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:04.186734 (+ 2924us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.3:49299: BlockingRecv error: recv got EOF from 127.8.102.3:49299 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 26 15:01:04 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:01:04 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.1@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:01:04.953627 17822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:04.948703 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.1:35351)
0926 15:01:04.948844 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:04.948859 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:04.948942 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:04.949116 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:04.949117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:04.949280 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:04.949338 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:04.949340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:04.950055 (+ 715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:04.950056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:04.950627 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:04.950678 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:04.951076 (+ 398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:04.951077 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:04.951162 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:04.951694 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:04.951699 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:04.953528 (+ 1829us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":57,"threads_started":1}
W20260926 15:01:04.953693 17823 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:01:04.948803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:35351 (local address 127.8.102.62:41841)
0926 15:01:04.948973 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:04.948975 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:04.948986 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:01:04.949026 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:04.949027 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:04.949037 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:04.949097 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:04.949403 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:04.949932 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:04.950777 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:04.950977 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:04.951049 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:04.953615 (+ 2566us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.1:35351: BlockingRecv error: recv got EOF from 127.8.102.1:35351 (error 108)
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":64,"threads_started":1}
Sep 26 15:01:05 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
Sep 26 15:01:05 dist-test-slave-jnhs krb5kdc[16763](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.8.102.2@KRBTEST.COM for oryx/127.8.102.62@KRBTEST.COM, Server not found in Kerberos database
I20260926 15:01:05.067699 17824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:05.062436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41841 (local address 127.8.102.2:51707)
0926 15:01:05.062584 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:05.062602 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:05.062708 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:05.062975 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:05.062977 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:05.063164 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:05.063215 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:05.063217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:05.063921 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:05.063921 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:05.064470 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:05.064472 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:05.064895 (+ 423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:05.064897 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:05.064997 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:05.065609 (+ 612us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:05.065614 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:05.067622 (+ 2008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.8.102.62:41841: Server oryx/127.8.102.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
W20260926 15:01:05.067808 17823 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:01:05.062560 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:51707 (local address 127.8.102.62:41841)
0926 15:01:05.062656 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:05.062658 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:05.062686 (+ 28us) server_negotiation.cc:407] Connection header received
0926 15:01:05.062837 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:05.062838 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:05.062851 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:05.062916 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:05.063301 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:05.063817 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:05.064575 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:05.064765 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:05.064862 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:05.067719 (+ 2857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.8.102.2:51707: BlockingRecv error: recv got EOF from 127.8.102.2:51707 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 15:01:05.108919 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 17346
I20260926 15:01:05.113662 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 17499
I20260926 15:01:05.118448 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 17632
I20260926 15:01:05.122851 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 17276
2026-09-26T15:01:05Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17549 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-26T15:01:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:01:05Z Disabled control of system clock
I20260926 15:01:05.151201 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:39165
--webserver_interface=127.8.102.62
--webserver_port=0
--builtin_ntp_servers=127.8.102.20:38357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:39165
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260926 15:01:05.235525 17828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:01:05.235750 17828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:01:05.235771 17828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:01:05.237033 17828 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:01:05.237079 17828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:01:05.237092 17828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:01:05.237104 17828 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:01:05.237116 17828 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:01:05.238469 17828 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:38357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:39165
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:39165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17828
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:01:05.238766 17828 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:01:05.239039 17828 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:01:05.244410 17836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:05.244429 17833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:05.244438 17834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:05.244467 17828 server_base.cc:1034] running on GCE node
I20260926 15:01:05.245126 17828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:01:05.245461 17828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:01:05.246717 17828 hybrid_clock.cc:648] HybridClock initialized: now 1790434865246692 us; error 31 us; skew 500 ppm
I20260926 15:01:05.248008 17828 webserver.cc:460] Webserver started at http://127.8.102.62:42279/ using document root <none> and password file <none>
I20260926 15:01:05.248296 17828 fs_manager.cc:359] Metadata directory not provided
I20260926 15:01:05.248422 17828 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:01:05.248576 17828 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:01:05.249694 17828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "88343eedf2e84fec9797850df48ef557"
format_stamp: "Formatted at 2026-09-26 15:01:05 on dist-test-slave-jnhs"
server_key: "d7801953ff165863d251cc0e9e39b119"
server_key_iv: "67816d4223baed321e2f8e3544ee06b9"
server_key_version: "encryptionkey@0"
I20260926 15:01:05.250069 17828 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "88343eedf2e84fec9797850df48ef557"
format_stamp: "Formatted at 2026-09-26 15:01:05 on dist-test-slave-jnhs"
server_key: "d7801953ff165863d251cc0e9e39b119"
server_key_iv: "67816d4223baed321e2f8e3544ee06b9"
server_key_version: "encryptionkey@0"
I20260926 15:01:05.251526 17828 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:01:05.252518 17842 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:05.252713 17828 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:01:05.252826 17828 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "88343eedf2e84fec9797850df48ef557"
format_stamp: "Formatted at 2026-09-26 15:01:05 on dist-test-slave-jnhs"
server_key: "d7801953ff165863d251cc0e9e39b119"
server_key_iv: "67816d4223baed321e2f8e3544ee06b9"
server_key_version: "encryptionkey@0"
I20260926 15:01:05.252909 17828 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:01:05.277128 17828 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:01:05.277514 17828 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:01:05.277684 17828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:01:05.281926 17828 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:39165
I20260926 15:01:05.281967 17894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:39165 every 8 connection(s)
I20260926 15:01:05.282361 17828 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:01:05.283377 17895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:01:05.285742 17895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557: Bootstrap starting.
I20260926 15:01:05.286168 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 17828
I20260926 15:01:05.286265 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:01:05.286496 8600 external_mini_cluster.cc:1428] Setting key fdaa3379d53c7249f87be624b4139b33
I20260926 15:01:05.286501 17895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557: Neither blocks nor log segments found. Creating new log.
I20260926 15:01:05.286867 17895 log.cc:824] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557: Log is configured to *not* fsync() on all Append() calls
I20260926 15:01:05.287683 17895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557: No bootstrap required, opened a new log
I20260926 15:01:05.289429 17895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88343eedf2e84fec9797850df48ef557" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 39165 } }
I20260926 15:01:05.289613 17895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:01:05.289647 17895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88343eedf2e84fec9797850df48ef557, State: Initialized, Role: FOLLOWER
I20260926 15:01:05.289772 17895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88343eedf2e84fec9797850df48ef557" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 39165 } }
I20260926 15:01:05.289850 17895 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:01:05.289880 17895 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:01:05.289925 17895 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:01:05.291025 17895 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88343eedf2e84fec9797850df48ef557" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 39165 } }
I20260926 15:01:05.291170 17895 leader_election.cc:302] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88343eedf2e84fec9797850df48ef557; no voters:
I20260926 15:01:05.291358 17895 leader_election.cc:288] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:01:05.291415 17900 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:01:05.291596 17900 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557 [term 1 LEADER]: Becoming Leader. State: Replica: 88343eedf2e84fec9797850df48ef557, State: Running, Role: LEADER
I20260926 15:01:05.291692 17895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:01:05.291731 17900 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88343eedf2e84fec9797850df48ef557" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 39165 } }
I20260926 15:01:05.292189 17901 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88343eedf2e84fec9797850df48ef557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88343eedf2e84fec9797850df48ef557" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 39165 } } }
I20260926 15:01:05.292265 17901 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557 [sys.catalog]: This master's current role is: LEADER
I20260926 15:01:05.292513 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:05.287084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51018 (local address 127.8.102.62:39165)
0926 15:01:05.287284 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:05.287292 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:05.287318 (+ 26us) server_negotiation.cc:407] Connection header received
0926 15:01:05.287769 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:05.287779 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:05.287862 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:05.287996 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:05.288236 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:05.289131 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:05.290219 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:05.290702 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:05.290811 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:05.291679 (+ 868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:01:05.291683 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:01:05.291683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:01:05.291921 (+ 238us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:01:05.291969 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:01:05.291985 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:01:05.292027 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:05.292170 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 15:01:05.292261 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":65,"threads_started":1}
I20260926 15:01:05.292510 17902 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88343eedf2e84fec9797850df48ef557. Latest consensus state: current_term: 1 leader_uuid: "88343eedf2e84fec9797850df48ef557" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88343eedf2e84fec9797850df48ef557" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 39165 } } }
I20260926 15:01:05.293183 17902 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557 [sys.catalog]: This master's current role is: LEADER
W20260926 15:01:05.293533 17916 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:01:05.293638 17916 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:01:05.293701 17907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:01:05.293848 17907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:01:05.295364 17907 catalog_manager.cc:1347] Generated new cluster ID: fda7a12c60fc41738a606478e31aa985
I20260926 15:01:05.295426 17907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:01:05.309652 17907 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:01:05.310237 17907 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:01:05.315073 17907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88343eedf2e84fec9797850df48ef557: Generated new TSK 0
I20260926 15:01:05.315284 17907 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:01:05.325989 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:39165
--builtin_ntp_servers=127.8.102.20:38357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 15:01:05.418752 17919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:01:05.418995 17919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:01:05.419029 17919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:01:05.420248 17919 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:01:05.420323 17919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:01:05.420365 17919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:01:05.421751 17919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:38357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:39165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17919
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:01:05.422044 17919 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:01:05.422356 17919 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:01:05.427830 17925 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:05.428025 17924 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:05.428146 17919 server_base.cc:1034] running on GCE node
W20260926 15:01:05.427819 17927 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:05.428455 17919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:01:05.428751 17919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:01:05.430042 17919 hybrid_clock.cc:648] HybridClock initialized: now 1790434865430032 us; error 54 us; skew 500 ppm
I20260926 15:01:05.431564 17919 webserver.cc:460] Webserver started at http://127.8.102.1:32885/ using document root <none> and password file <none>
I20260926 15:01:05.431820 17919 fs_manager.cc:359] Metadata directory not provided
I20260926 15:01:05.431867 17919 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:01:05.431972 17919 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:01:05.433103 17919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "f825d57730a04bc887d105ee4e8f8965"
format_stamp: "Formatted at 2026-09-26 15:01:05 on dist-test-slave-jnhs"
server_key: "5a951259a928c45b1ec48ee2b9ae1e6f"
server_key_iv: "6b908212eca2f30a5b90500e110b37d5"
server_key_version: "encryptionkey@0"
I20260926 15:01:05.433470 17919 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "f825d57730a04bc887d105ee4e8f8965"
format_stamp: "Formatted at 2026-09-26 15:01:05 on dist-test-slave-jnhs"
server_key: "5a951259a928c45b1ec48ee2b9ae1e6f"
server_key_iv: "6b908212eca2f30a5b90500e110b37d5"
server_key_version: "encryptionkey@0"
I20260926 15:01:05.434867 17919 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:01:05.435698 17933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:05.435853 17919 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:01:05.435966 17919 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "f825d57730a04bc887d105ee4e8f8965"
format_stamp: "Formatted at 2026-09-26 15:01:05 on dist-test-slave-jnhs"
server_key: "5a951259a928c45b1ec48ee2b9ae1e6f"
server_key_iv: "6b908212eca2f30a5b90500e110b37d5"
server_key_version: "encryptionkey@0"
I20260926 15:01:05.436040 17919 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:01:05.445191 17919 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:01:05.445508 17919 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:01:05.445653 17919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:01:05.445881 17919 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:01:05.446201 17919 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:01:05.446234 17919 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:05.446262 17919 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:01:05.446312 17919 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:05.452492 17919 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:44547
I20260926 15:01:05.452543 18046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:44547 every 8 connection(s)
I20260926 15:01:05.452875 17919 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:01:05.458531 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:05.453654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:55127 (local address 127.8.102.62:39165)
0926 15:01:05.453713 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:05.453716 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:05.453942 (+ 226us) server_negotiation.cc:407] Connection header received
0926 15:01:05.454477 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:05.454478 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:05.454487 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:05.454550 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:05.455329 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:05.455941 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:05.457133 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:05.457325 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:05.457376 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:05.458277 (+ 901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:01:05.458278 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:01:05.458278 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:01:05.458283 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:01:05.458298 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:01:05.458302 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:01:05.458345 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:05.458434 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 15:01:05.458456 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260926 15:01:05.458775 18049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:05.453609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:39165 (local address 127.8.102.1:55127)
0926 15:01:05.453800 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:05.453842 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:05.454354 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:05.454628 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:05.454677 (+ 49us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:05.454689 (+ 12us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:05.455225 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:05.455230 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:05.456109 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:05.456110 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:05.457033 (+ 923us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:05.457036 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:05.457430 (+ 394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:05.457431 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:05.457569 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:05.458137 (+ 568us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:01:05.458137 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:05.458168 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:01:05.458194 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:01:05.458361 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:01:05.458365 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:01:05.458367 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 15:01:05.458425 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 15:01:05.458514 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":93,"threads_started":1}
I20260926 15:01:05.459326 18047 heartbeater.cc:347] Connected to a master server at 127.8.102.62:39165
I20260926 15:01:05.459450 18047 heartbeater.cc:464] Registering TS with master...
I20260926 15:01:05.459679 18047 heartbeater.cc:511] Master 127.8.102.62:39165 requested a full tablet report, sending...
I20260926 15:01:05.460225 17859 ts_manager.cc:194] Registered new tserver with Master: f825d57730a04bc887d105ee4e8f8965 (127.8.102.1:44547)
I20260926 15:01:05.460901 17859 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.102.1:55127
I20260926 15:01:05.461998 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 17919
I20260926 15:01:05.462077 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:01:05.462283 8600 external_mini_cluster.cc:1428] Setting key 70bf38738302ee7134eea4c893843445
I20260926 15:01:05.463336 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:39165
--builtin_ntp_servers=127.8.102.20:38357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 15:01:05.547158 18050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:01:05.547375 18050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:01:05.547398 18050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:01:05.548604 18050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:01:05.548647 18050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:01:05.548672 18050 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:01:05.550050 18050 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:38357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:39165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:01:05.550324 18050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:01:05.550709 18050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:01:05.556046 18056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:05.556233 18050 server_base.cc:1034] running on GCE node
W20260926 15:01:05.556026 18055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:05.556030 18058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:05.556769 18050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:01:05.557040 18050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:01:05.558202 18050 hybrid_clock.cc:648] HybridClock initialized: now 1790434865558199 us; error 37 us; skew 500 ppm
I20260926 15:01:05.559664 18050 webserver.cc:460] Webserver started at http://127.8.102.2:43489/ using document root <none> and password file <none>
I20260926 15:01:05.559937 18050 fs_manager.cc:359] Metadata directory not provided
I20260926 15:01:05.559990 18050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:01:05.560168 18050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:01:05.561259 18050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "71f7ecfc245944069587ad57a37d0bd1"
format_stamp: "Formatted at 2026-09-26 15:01:05 on dist-test-slave-jnhs"
server_key: "0952931313a97b9aacefcf367f9a23c9"
server_key_iv: "8ccf3521bc17b46679590dbefc1762db"
server_key_version: "encryptionkey@0"
I20260926 15:01:05.561628 18050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "71f7ecfc245944069587ad57a37d0bd1"
format_stamp: "Formatted at 2026-09-26 15:01:05 on dist-test-slave-jnhs"
server_key: "0952931313a97b9aacefcf367f9a23c9"
server_key_iv: "8ccf3521bc17b46679590dbefc1762db"
server_key_version: "encryptionkey@0"
I20260926 15:01:05.563128 18050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 15:01:05.564023 18064 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:05.564291 18050 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:01:05.564452 18050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "71f7ecfc245944069587ad57a37d0bd1"
format_stamp: "Formatted at 2026-09-26 15:01:05 on dist-test-slave-jnhs"
server_key: "0952931313a97b9aacefcf367f9a23c9"
server_key_iv: "8ccf3521bc17b46679590dbefc1762db"
server_key_version: "encryptionkey@0"
I20260926 15:01:05.564549 18050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:01:05.584759 18050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:01:05.585124 18050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:01:05.585292 18050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:01:05.585598 18050 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:01:05.585956 18050 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:01:05.586018 18050 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:05.586053 18050 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:01:05.586097 18050 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:05.592124 18050 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:41259
I20260926 15:01:05.592172 18177 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:41259 every 8 connection(s)
I20260926 15:01:05.592573 18050 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:01:05.598095 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:05.593400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:51473 (local address 127.8.102.62:39165)
0926 15:01:05.593511 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:05.593513 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:05.593670 (+ 157us) server_negotiation.cc:407] Connection header received
0926 15:01:05.594234 (+ 564us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:05.594235 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:05.594247 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:05.594341 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:05.595127 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:05.595639 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:05.596679 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:05.596862 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:05.596918 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:05.597826 (+ 908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:01:05.597827 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:01:05.597827 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:01:05.597832 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:01:05.597847 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:01:05.597851 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:01:05.597898 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:05.598002 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 15:01:05.598028 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260926 15:01:05.598326 18180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:05.593326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:39165 (local address 127.8.102.2:51473)
0926 15:01:05.593527 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:05.593563 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:05.594120 (+ 557us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:05.594426 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:05.594430 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:05.594439 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:05.595021 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:05.595026 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:05.595728 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:05.595729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:05.596592 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:05.596594 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:05.596943 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:05.596944 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:05.597060 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:05.597654 (+ 594us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:01:05.597654 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:05.597685 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:01:05.597711 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:01:05.597912 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:01:05.597916 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:01:05.597919 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:01:05.597974 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 15:01:05.598064 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":71,"threads_started":1}
I20260926 15:01:05.598796 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 18050
I20260926 15:01:05.598904 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:01:05.598912 18178 heartbeater.cc:347] Connected to a master server at 127.8.102.62:39165
I20260926 15:01:05.599012 18178 heartbeater.cc:464] Registering TS with master...
I20260926 15:01:05.599113 8600 external_mini_cluster.cc:1428] Setting key 2378b939398351b086c5e51c55b009e3
I20260926 15:01:05.599224 18178 heartbeater.cc:511] Master 127.8.102.62:39165 requested a full tablet report, sending...
I20260926 15:01:05.599606 17858 ts_manager.cc:194] Registered new tserver with Master: 71f7ecfc245944069587ad57a37d0bd1 (127.8.102.2:41259)
I20260926 15:01:05.599984 17858 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.102.2:51473
I20260926 15:01:05.600118 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:39165
--builtin_ntp_servers=127.8.102.20:38357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260926 15:01:05.683805 18181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:01:05.684067 18181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:01:05.684091 18181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:01:05.685397 18181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:01:05.685451 18181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:01:05.685480 18181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:01:05.686864 18181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:38357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:39165
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18181
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:01:05.687085 18181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:01:05.687369 18181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:01:05.692595 18187 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:05.692692 18189 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:05.692793 18181 server_base.cc:1034] running on GCE node
W20260926 15:01:05.692740 18186 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:05.693198 18181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:01:05.693496 18181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:01:05.694693 18181 hybrid_clock.cc:648] HybridClock initialized: now 1790434865694679 us; error 37 us; skew 500 ppm
I20260926 15:01:05.696055 18181 webserver.cc:460] Webserver started at http://127.8.102.3:36809/ using document root <none> and password file <none>
I20260926 15:01:05.696451 18181 fs_manager.cc:359] Metadata directory not provided
I20260926 15:01:05.696525 18181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:01:05.696699 18181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:01:05.697831 18181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "98af852d0a064214bd42325c16c13b40"
format_stamp: "Formatted at 2026-09-26 15:01:05 on dist-test-slave-jnhs"
server_key: "e787528ecb048100898d9d3aa3f1483e"
server_key_iv: "48fae37d39befb53f0adf04df5d832e8"
server_key_version: "encryptionkey@0"
I20260926 15:01:05.698201 18181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "98af852d0a064214bd42325c16c13b40"
format_stamp: "Formatted at 2026-09-26 15:01:05 on dist-test-slave-jnhs"
server_key: "e787528ecb048100898d9d3aa3f1483e"
server_key_iv: "48fae37d39befb53f0adf04df5d832e8"
server_key_version: "encryptionkey@0"
I20260926 15:01:05.699748 18181 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:01:05.700593 18195 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:05.700845 18181 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:01:05.700958 18181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "98af852d0a064214bd42325c16c13b40"
format_stamp: "Formatted at 2026-09-26 15:01:05 on dist-test-slave-jnhs"
server_key: "e787528ecb048100898d9d3aa3f1483e"
server_key_iv: "48fae37d39befb53f0adf04df5d832e8"
server_key_version: "encryptionkey@0"
I20260926 15:01:05.701043 18181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:01:05.717347 18181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:01:05.717716 18181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:01:05.717875 18181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:01:05.718113 18181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:01:05.718600 18181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:01:05.718710 18181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:05.718743 18181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:01:05.718760 18181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:05.725189 18181 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:38733
I20260926 15:01:05.725282 18308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:38733 every 8 connection(s)
I20260926 15:01:05.725669 18181 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:01:05.731245 17899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:05.726519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:50627 (local address 127.8.102.62:39165)
0926 15:01:05.726603 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:05.726606 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:05.726857 (+ 251us) server_negotiation.cc:407] Connection header received
0926 15:01:05.727420 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:05.727422 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:05.727433 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:05.727499 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:05.728250 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:05.728755 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:05.729775 (+ 1020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:05.729977 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:05.730044 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:05.730993 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:01:05.730995 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:01:05.730995 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:01:05.731000 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:01:05.731015 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:01:05.731019 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:01:05.731068 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:05.731156 (+ 88us) server_negotiation.cc:299] Negotiation successful
0926 15:01:05.731180 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:01:05.731444 18311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:05.726430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:39165 (local address 127.8.102.3:50627)
0926 15:01:05.726769 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:05.726802 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:05.727272 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:05.727591 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:05.727595 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:05.727603 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:05.728135 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:05.728139 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:05.728843 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:05.728844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:05.729664 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:05.729667 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:05.730095 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:05.730096 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:05.730226 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:05.730847 (+ 621us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:01:05.730848 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:05.730879 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:01:05.730905 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:01:05.731082 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:01:05.731087 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:01:05.731090 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:01:05.731146 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 15:01:05.731233 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"thread_start_us":81,"threads_started":1}
I20260926 15:01:05.732048 18309 heartbeater.cc:347] Connected to a master server at 127.8.102.62:39165
I20260926 15:01:05.732157 18309 heartbeater.cc:464] Registering TS with master...
I20260926 15:01:05.732314 18309 heartbeater.cc:511] Master 127.8.102.62:39165 requested a full tablet report, sending...
I20260926 15:01:05.732679 17858 ts_manager.cc:194] Registered new tserver with Master: 98af852d0a064214bd42325c16c13b40 (127.8.102.3:38733)
I20260926 15:01:05.733075 17858 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.102.3:50627
I20260926 15:01:05.735517 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 18181
I20260926 15:01:05.735603 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:01:05.735770 8600 external_mini_cluster.cc:1428] Setting key cdad78a4e12eab2aa3a7b71089db6214
I20260926 15:01:05.736974 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 15:01:05.739409 17899 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:01:05.738728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51034 (local address 127.8.102.62:39165)
0926 15:01:05.738825 (+ 97us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:05.738828 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:05.738937 (+ 109us) server_negotiation.cc:407] Connection header received
0926 15:01:05.739028 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:05.739029 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:05.739040 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:05.739092 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:05.739285 (+ 193us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51034: BlockingRecv error: recv got EOF from 127.0.0.1:51034 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 15:01:05.739500 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 17919
I20260926 15:01:05.744632 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 18050
I20260926 15:01:05.749035 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 18181
I20260926 15:01:05.753264 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 17828
2026-09-26T15:01:05Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (629 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-26T15:01:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:01:05Z Disabled control of system clock
I20260926 15:01:05.781592 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:41487
--webserver_interface=127.8.102.62
--webserver_port=0
--builtin_ntp_servers=127.8.102.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:41487
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 15:01:05.867220 18320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:01:05.867478 18320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:01:05.867517 18320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:01:05.868693 18320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:01:05.868763 18320 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:01:05.868785 18320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:01:05.868808 18320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:01:05.868837 18320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:01:05.870221 18320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:41487
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:41487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18320
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:01:05.870522 18320 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:01:05.870851 18320 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:01:05.875867 18325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:05.875865 18328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:05.875869 18326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:05.876214 18320 server_base.cc:1034] running on GCE node
I20260926 15:01:05.876564 18320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:01:05.876860 18320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:01:05.878029 18320 hybrid_clock.cc:648] HybridClock initialized: now 1790434865878022 us; error 55 us; skew 500 ppm
I20260926 15:01:05.879320 18320 webserver.cc:460] Webserver started at http://127.8.102.62:38189/ using document root <none> and password file <none>
I20260926 15:01:05.879580 18320 fs_manager.cc:359] Metadata directory not provided
I20260926 15:01:05.879648 18320 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:01:05.879801 18320 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:01:05.880930 18320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "04355a879d6e42db9127d0270ee5a34d"
format_stamp: "Formatted at 2026-09-26 15:01:05 on dist-test-slave-jnhs"
server_key: "e8d129330dd4dd5841376cb22ecfb24b"
server_key_iv: "928ce3c8ad9e5b5e98d2ffb7e0ffc4ae"
server_key_version: "encryptionkey@0"
I20260926 15:01:05.881294 18320 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "04355a879d6e42db9127d0270ee5a34d"
format_stamp: "Formatted at 2026-09-26 15:01:05 on dist-test-slave-jnhs"
server_key: "e8d129330dd4dd5841376cb22ecfb24b"
server_key_iv: "928ce3c8ad9e5b5e98d2ffb7e0ffc4ae"
server_key_version: "encryptionkey@0"
I20260926 15:01:05.882764 18320 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 15:01:05.883752 18334 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:05.884023 18320 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:01:05.884136 18320 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "04355a879d6e42db9127d0270ee5a34d"
format_stamp: "Formatted at 2026-09-26 15:01:05 on dist-test-slave-jnhs"
server_key: "e8d129330dd4dd5841376cb22ecfb24b"
server_key_iv: "928ce3c8ad9e5b5e98d2ffb7e0ffc4ae"
server_key_version: "encryptionkey@0"
I20260926 15:01:05.884229 18320 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:01:05.887928 18320 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:01:05.888239 18320 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:01:05.888381 18320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:01:05.892812 18320 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:41487
I20260926 15:01:05.892844 18386 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:41487 every 8 connection(s)
I20260926 15:01:05.893291 18320 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:01:05.894132 18387 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:01:05.896453 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 18320
I20260926 15:01:05.896589 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:01:05.896777 8600 external_mini_cluster.cc:1428] Setting key c2fb031927fef7726b1d469804e59861
I20260926 15:01:05.898052 18387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d: Bootstrap starting.
I20260926 15:01:05.898911 18387 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d: Neither blocks nor log segments found. Creating new log.
I20260926 15:01:05.899287 18387 log.cc:824] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d: Log is configured to *not* fsync() on all Append() calls
I20260926 15:01:05.899624 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:05.897417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37190 (local address 127.8.102.62:41487)
0926 15:01:05.897606 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:05.897616 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:05.897641 (+ 25us) server_negotiation.cc:407] Connection header received
0926 15:01:05.898083 (+ 442us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:05.898092 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:05.898136 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:05.898304 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:05.899066 (+ 762us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:01:05.899070 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:01:05.899070 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:01:05.899074 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:01:05.899131 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:01:05.899144 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:01:05.899224 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:05.899310 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 15:01:05.899446 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110,"thread_start_us":81,"threads_started":1}
I20260926 15:01:05.900153 18387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d: No bootstrap required, opened a new log
I20260926 15:01:05.901597 18387 raft_consensus.cc:348] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04355a879d6e42db9127d0270ee5a34d" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41487 } }
I20260926 15:01:05.901760 18387 raft_consensus.cc:374] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:01:05.901808 18387 raft_consensus.cc:729] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 04355a879d6e42db9127d0270ee5a34d, State: Initialized, Role: FOLLOWER
I20260926 15:01:05.901962 18387 consensus_queue.cc:260] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04355a879d6e42db9127d0270ee5a34d" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41487 } }
I20260926 15:01:05.902055 18387 raft_consensus.cc:388] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:01:05.902098 18387 raft_consensus.cc:482] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:01:05.902149 18387 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:01:05.903090 18387 raft_consensus.cc:504] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04355a879d6e42db9127d0270ee5a34d" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41487 } }
I20260926 15:01:05.903216 18387 leader_election.cc:302] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 04355a879d6e42db9127d0270ee5a34d; no voters:
I20260926 15:01:05.903401 18387 leader_election.cc:288] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:01:05.903530 18392 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:01:05.903671 18392 raft_consensus.cc:686] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d [term 1 LEADER]: Becoming Leader. State: Replica: 04355a879d6e42db9127d0270ee5a34d, State: Running, Role: LEADER
I20260926 15:01:05.903692 18387 sys_catalog.cc:564] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:01:05.903779 18392 consensus_queue.cc:237] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04355a879d6e42db9127d0270ee5a34d" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41487 } }
I20260926 15:01:05.904217 18394 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 04355a879d6e42db9127d0270ee5a34d. Latest consensus state: current_term: 1 leader_uuid: "04355a879d6e42db9127d0270ee5a34d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04355a879d6e42db9127d0270ee5a34d" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41487 } } }
I20260926 15:01:05.904229 18393 sys_catalog.cc:455] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "04355a879d6e42db9127d0270ee5a34d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "04355a879d6e42db9127d0270ee5a34d" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41487 } } }
I20260926 15:01:05.904421 18394 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d [sys.catalog]: This master's current role is: LEADER
I20260926 15:01:05.904434 18393 sys_catalog.cc:458] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d [sys.catalog]: This master's current role is: LEADER
I20260926 15:01:05.905211 18401 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:01:05.905793 18401 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:01:05.907310 18401 catalog_manager.cc:1347] Generated new cluster ID: 7eac1302692e43a7a84723871c61f55b
I20260926 15:01:05.907363 18401 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:01:05.920475 18401 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:01:05.920710 18401 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:01:05.926331 18401 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 04355a879d6e42db9127d0270ee5a34d: Generated new TSK 0
I20260926 15:01:05.926561 18401 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:01:05.931222 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:41487
--builtin_ntp_servers=127.8.102.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260926 15:01:06.018478 18411 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:01:06.018778 18411 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:01:06.018807 18411 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:01:06.020082 18411 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:01:06.020128 18411 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:01:06.020156 18411 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:01:06.021590 18411 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:41487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18411
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:01:06.021873 18411 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:01:06.022207 18411 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:01:06.027410 18416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:06.027534 18411 server_base.cc:1034] running on GCE node
W20260926 15:01:06.027484 18419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:06.027417 18417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:06.027860 18411 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:01:06.028134 18411 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:01:06.029292 18411 hybrid_clock.cc:648] HybridClock initialized: now 1790434866029279 us; error 37 us; skew 500 ppm
I20260926 15:01:06.031327 18411 webserver.cc:460] Webserver started at http://127.8.102.1:43155/ using document root <none> and password file <none>
I20260926 15:01:06.031636 18411 fs_manager.cc:359] Metadata directory not provided
I20260926 15:01:06.031702 18411 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:01:06.031857 18411 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:01:06.033470 18411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "158ddd18facf4e2493bb1b8389b5b246"
format_stamp: "Formatted at 2026-09-26 15:01:06 on dist-test-slave-jnhs"
server_key: "0f249645970088fb3d39516b0c34ccaa"
server_key_iv: "1b7a982ec41a6425de8d1f15b9aa4459"
server_key_version: "encryptionkey@0"
I20260926 15:01:06.033932 18411 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "158ddd18facf4e2493bb1b8389b5b246"
format_stamp: "Formatted at 2026-09-26 15:01:06 on dist-test-slave-jnhs"
server_key: "0f249645970088fb3d39516b0c34ccaa"
server_key_iv: "1b7a982ec41a6425de8d1f15b9aa4459"
server_key_version: "encryptionkey@0"
I20260926 15:01:06.035924 18411 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.003s
I20260926 15:01:06.037021 18425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:06.037253 18411 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:01:06.037361 18411 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "158ddd18facf4e2493bb1b8389b5b246"
format_stamp: "Formatted at 2026-09-26 15:01:06 on dist-test-slave-jnhs"
server_key: "0f249645970088fb3d39516b0c34ccaa"
server_key_iv: "1b7a982ec41a6425de8d1f15b9aa4459"
server_key_version: "encryptionkey@0"
I20260926 15:01:06.037462 18411 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:01:06.043393 18411 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:01:06.043747 18411 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:01:06.043916 18411 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:01:06.044169 18411 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:01:06.044605 18411 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:01:06.044647 18411 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:06.044672 18411 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:01:06.044723 18411 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:06.052531 18411 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:33811
I20260926 15:01:06.052578 18538 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:33811 every 8 connection(s)
I20260926 15:01:06.052964 18411 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:01:06.056422 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 18411
I20260926 15:01:06.056581 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:01:06.056834 8600 external_mini_cluster.cc:1428] Setting key 250ebc6fbd2aa2d117137b41261ee680
I20260926 15:01:06.056845 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:06.053846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:55231 (local address 127.8.102.62:41487)
0926 15:01:06.053927 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:06.053929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:06.054231 (+ 302us) server_negotiation.cc:407] Connection header received
0926 15:01:06.054924 (+ 693us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:06.054925 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:06.054931 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:06.055004 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:06.056563 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:01:06.056565 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:01:06.056566 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:01:06.056572 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:01:06.056593 (+ 21us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:01:06.056598 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:01:06.056669 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:06.056755 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 15:01:06.056773 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:01:06.057143 18541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:06.053774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41487 (local address 127.8.102.1:55231)
0926 15:01:06.054030 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:06.054074 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:06.054756 (+ 682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:06.055127 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:06.055132 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:06.055142 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:06.056349 (+ 1207us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:01:06.056350 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:06.056393 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:01:06.056428 (+ 35us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:01:06.056668 (+ 240us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:01:06.056674 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:01:06.056678 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:01:06.056756 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 15:01:06.056820 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"thread_start_us":104,"threads_started":1}
I20260926 15:01:06.057832 18539 heartbeater.cc:347] Connected to a master server at 127.8.102.62:41487
I20260926 15:01:06.057945 18539 heartbeater.cc:464] Registering TS with master...
I20260926 15:01:06.058112 18539 heartbeater.cc:511] Master 127.8.102.62:41487 requested a full tablet report, sending...
I20260926 15:01:06.058113 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:41487
--builtin_ntp_servers=127.8.102.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 15:01:06.058784 18350 ts_manager.cc:194] Registered new tserver with Master: 158ddd18facf4e2493bb1b8389b5b246 (127.8.102.1:33811)
W20260926 15:01:06.144022 18542 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:01:06.144282 18542 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:01:06.144321 18542 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:01:06.145553 18542 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:01:06.145622 18542 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:01:06.145666 18542 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:01:06.147111 18542 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:41487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18542
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:01:06.147389 18542 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:01:06.147707 18542 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:01:06.152840 18548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:06.152818 18550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:06.152805 18547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:06.153267 18542 server_base.cc:1034] running on GCE node
I20260926 15:01:06.153542 18542 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:01:06.153757 18542 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:01:06.154927 18542 hybrid_clock.cc:648] HybridClock initialized: now 1790434866154899 us; error 40 us; skew 500 ppm
I20260926 15:01:06.156440 18542 webserver.cc:460] Webserver started at http://127.8.102.2:37445/ using document root <none> and password file <none>
I20260926 15:01:06.156653 18542 fs_manager.cc:359] Metadata directory not provided
I20260926 15:01:06.156698 18542 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:01:06.156795 18542 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:01:06.157938 18542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "33af515fad7a4a16956e589653d2664d"
format_stamp: "Formatted at 2026-09-26 15:01:06 on dist-test-slave-jnhs"
server_key: "d5ce51fab557d837d652617e34b52b4b"
server_key_iv: "1758100c62bdd52912ed5207b6467402"
server_key_version: "encryptionkey@0"
I20260926 15:01:06.158257 18542 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "33af515fad7a4a16956e589653d2664d"
format_stamp: "Formatted at 2026-09-26 15:01:06 on dist-test-slave-jnhs"
server_key: "d5ce51fab557d837d652617e34b52b4b"
server_key_iv: "1758100c62bdd52912ed5207b6467402"
server_key_version: "encryptionkey@0"
I20260926 15:01:06.159708 18542 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 15:01:06.160632 18556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:06.160866 18542 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:01:06.160959 18542 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "33af515fad7a4a16956e589653d2664d"
format_stamp: "Formatted at 2026-09-26 15:01:06 on dist-test-slave-jnhs"
server_key: "d5ce51fab557d837d652617e34b52b4b"
server_key_iv: "1758100c62bdd52912ed5207b6467402"
server_key_version: "encryptionkey@0"
I20260926 15:01:06.161027 18542 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:01:06.164667 18542 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:01:06.164906 18542 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:01:06.165002 18542 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:01:06.165156 18542 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:01:06.165407 18542 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:01:06.165436 18542 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:06.165457 18542 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:01:06.165470 18542 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:06.171423 18542 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:33061
I20260926 15:01:06.171542 18669 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:33061 every 8 connection(s)
I20260926 15:01:06.171851 18542 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:01:06.173254 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 18542
I20260926 15:01:06.173444 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:01:06.173641 8600 external_mini_cluster.cc:1428] Setting key ffe47bd09f7df21dfc784b541e9f0161
I20260926 15:01:06.174860 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:41487
--builtin_ntp_servers=127.8.102.20:36715
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260926 15:01:06.175374 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:06.172657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:44393 (local address 127.8.102.62:41487)
0926 15:01:06.172737 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:06.172738 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:06.172996 (+ 258us) server_negotiation.cc:407] Connection header received
0926 15:01:06.173583 (+ 587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:06.173585 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:06.173591 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:06.173647 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:06.175128 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:01:06.175129 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:01:06.175129 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:01:06.175135 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:01:06.175149 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:01:06.175153 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:01:06.175209 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:06.175289 (+ 80us) server_negotiation.cc:299] Negotiation successful
0926 15:01:06.175304 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260926 15:01:06.175614 18672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:06.172650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41487 (local address 127.8.102.2:44393)
0926 15:01:06.172847 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:06.172891 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:06.173483 (+ 592us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:06.173722 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:06.173726 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:06.173737 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:06.174917 (+ 1180us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:01:06.174918 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:06.174965 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:01:06.175010 (+ 45us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:01:06.175218 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:01:06.175222 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:01:06.175226 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:01:06.175300 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 15:01:06.175363 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260926 15:01:06.176072 18670 heartbeater.cc:347] Connected to a master server at 127.8.102.62:41487
I20260926 15:01:06.176164 18670 heartbeater.cc:464] Registering TS with master...
I20260926 15:01:06.176330 18670 heartbeater.cc:511] Master 127.8.102.62:41487 requested a full tablet report, sending...
I20260926 15:01:06.176731 18350 ts_manager.cc:194] Registered new tserver with Master: 33af515fad7a4a16956e589653d2664d (127.8.102.2:33061)
W20260926 15:01:06.257783 18673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:01:06.258021 18673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:01:06.258045 18673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:01:06.259413 18673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:01:06.259467 18673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:01:06.259497 18673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:01:06.260933 18673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36715
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:41487
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:01:06.261145 18673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:01:06.261402 18673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:01:06.266677 18681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:06.266724 18679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:06.266690 18678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:06.267019 18673 server_base.cc:1034] running on GCE node
I20260926 15:01:06.267205 18673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:01:06.267509 18673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:01:06.268666 18673 hybrid_clock.cc:648] HybridClock initialized: now 1790434866268655 us; error 28 us; skew 500 ppm
I20260926 15:01:06.270078 18673 webserver.cc:460] Webserver started at http://127.8.102.3:36517/ using document root <none> and password file <none>
I20260926 15:01:06.270323 18673 fs_manager.cc:359] Metadata directory not provided
I20260926 15:01:06.270428 18673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:01:06.270589 18673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:01:06.271705 18673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "e31a6861ae204cdfb8296ac496fc1f3d"
format_stamp: "Formatted at 2026-09-26 15:01:06 on dist-test-slave-jnhs"
server_key: "7959022570aaa509c3859a6dd3da2c3a"
server_key_iv: "938995cd8a046913c7a0560b7778019e"
server_key_version: "encryptionkey@0"
I20260926 15:01:06.272096 18673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "e31a6861ae204cdfb8296ac496fc1f3d"
format_stamp: "Formatted at 2026-09-26 15:01:06 on dist-test-slave-jnhs"
server_key: "7959022570aaa509c3859a6dd3da2c3a"
server_key_iv: "938995cd8a046913c7a0560b7778019e"
server_key_version: "encryptionkey@0"
I20260926 15:01:06.273528 18673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 15:01:06.274335 18687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:06.274574 18673 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:01:06.274698 18673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "e31a6861ae204cdfb8296ac496fc1f3d"
format_stamp: "Formatted at 2026-09-26 15:01:06 on dist-test-slave-jnhs"
server_key: "7959022570aaa509c3859a6dd3da2c3a"
server_key_iv: "938995cd8a046913c7a0560b7778019e"
server_key_version: "encryptionkey@0"
I20260926 15:01:06.274907 18673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:01:06.278442 18673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:01:06.278785 18673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:01:06.278945 18673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:01:06.279156 18673 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:01:06.279539 18673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:01:06.279596 18673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:06.279636 18673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:01:06.279664 18673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:06.285662 18673 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:39463
I20260926 15:01:06.285717 18800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:39463 every 8 connection(s)
I20260926 15:01:06.286082 18673 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:01:06.289248 18390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:06.287045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:52913 (local address 127.8.102.62:41487)
0926 15:01:06.287088 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:06.287090 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:06.287205 (+ 115us) server_negotiation.cc:407] Connection header received
0926 15:01:06.287683 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:06.287685 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:06.287699 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:06.287822 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:06.289061 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:01:06.289062 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:01:06.289063 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:01:06.289066 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:01:06.289082 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:01:06.289085 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:01:06.289134 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:06.289181 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 15:01:06.289193 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 15:01:06.289522 18803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:06.286883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:41487 (local address 127.8.102.3:52913)
0926 15:01:06.287070 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:06.287113 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:06.287572 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:06.287987 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:06.287991 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:06.288000 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:06.288909 (+ 909us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:01:06.288910 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:06.288940 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:01:06.288970 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:01:06.289132 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:01:06.289134 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:01:06.289136 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 15:01:06.289188 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 15:01:06.289237 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":95,"threads_started":1}
I20260926 15:01:06.289996 18801 heartbeater.cc:347] Connected to a master server at 127.8.102.62:41487
I20260926 15:01:06.290081 18801 heartbeater.cc:464] Registering TS with master...
I20260926 15:01:06.290158 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 18673
I20260926 15:01:06.290269 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:01:06.290271 18801 heartbeater.cc:511] Master 127.8.102.62:41487 requested a full tablet report, sending...
I20260926 15:01:06.290475 8600 external_mini_cluster.cc:1428] Setting key 5373280f5a808f23e9afb047f9f00610
I20260926 15:01:06.290714 18350 ts_manager.cc:194] Registered new tserver with Master: e31a6861ae204cdfb8296ac496fc1f3d (127.8.102.3:39463)
I20260926 15:01:06.291352 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260926 15:01:06.293133 18390 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:01:06.292715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37204 (local address 127.8.102.62:41487)
0926 15:01:06.292750 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:06.292751 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:06.292866 (+ 115us) server_negotiation.cc:407] Connection header received
0926 15:01:06.292927 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:06.292927 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:06.292931 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:06.292968 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:06.293066 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37204: BlockingRecv error: recv got EOF from 127.0.0.1:37204 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 15:01:06.293304 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 18411
I20260926 15:01:06.298484 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 18542
I20260926 15:01:06.302770 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 18673
I20260926 15:01:06.307447 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 18320
2026-09-26T15:01:06Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (553 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:01:06 dist-test-slave-jnhs krb5kdc[18811](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:01:06 dist-test-slave-jnhs krb5kdc[18811](info): set up 2 sockets
Sep 26 15:01:06 dist-test-slave-jnhs krb5kdc[18811](info): commencing operation
krb5kdc: starting...
W20260926 15:01:07.387810 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:01:07 dist-test-slave-jnhs krb5kdc[18811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:01:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:01:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:01:07.522610 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:39521
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:39521
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:01:07.613142 18824 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:01:07.613480 18824 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:01:07.613533 18824 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:01:07.615020 18824 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:01:07.615123 18824 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:01:07.615149 18824 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:01:07.615172 18824 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:01:07.615203 18824 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:01:07.617055 18824 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:39521
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:39521
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18824
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:01:07.617404 18824 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:01:07.617684 18824 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:01:07.624060 18832 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:07.624085 18830 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:07.624060 18829 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:07.624437 18824 server_base.cc:1034] running on GCE node
I20260926 15:01:07.624747 18824 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:01:07.625119 18824 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:01:07.626302 18824 hybrid_clock.cc:648] HybridClock initialized: now 1790434867626275 us; error 40 us; skew 500 ppm
Sep 26 15:01:07 dist-test-slave-jnhs krb5kdc[18811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434867, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:01:07.629117 18824 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:01:07.629611 18824 webserver.cc:460] Webserver started at http://127.8.102.62:40943/ using document root <none> and password file <none>
I20260926 15:01:07.629827 18824 fs_manager.cc:359] Metadata directory not provided
I20260926 15:01:07.629871 18824 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:01:07.629978 18824 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:01:07.631114 18824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "d02c290eb0a543b19da8aabe77102c03"
format_stamp: "Formatted at 2026-09-26 15:01:07 on dist-test-slave-jnhs"
server_key: "81ed56ee91b669697dd5e7febb69246c"
server_key_iv: "37eabfb58fe8dffa50d10db41e657ca1"
server_key_version: "encryptionkey@0"
I20260926 15:01:07.631428 18824 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "d02c290eb0a543b19da8aabe77102c03"
format_stamp: "Formatted at 2026-09-26 15:01:07 on dist-test-slave-jnhs"
server_key: "81ed56ee91b669697dd5e7febb69246c"
server_key_iv: "37eabfb58fe8dffa50d10db41e657ca1"
server_key_version: "encryptionkey@0"
I20260926 15:01:07.632820 18824 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:01:07.633735 18839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:07.634142 18824 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:01:07.634367 18824 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "d02c290eb0a543b19da8aabe77102c03"
format_stamp: "Formatted at 2026-09-26 15:01:07 on dist-test-slave-jnhs"
server_key: "81ed56ee91b669697dd5e7febb69246c"
server_key_iv: "37eabfb58fe8dffa50d10db41e657ca1"
server_key_version: "encryptionkey@0"
I20260926 15:01:07.634506 18824 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:01:07.643111 18824 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:01:07.645977 18824 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:01:07.646163 18824 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:01:07.650390 18824 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:39521
I20260926 15:01:07.650429 18891 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:39521 every 8 connection(s)
I20260926 15:01:07.650866 18824 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:01:07.651823 18892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:01:07.654300 18892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03: Bootstrap starting.
I20260926 15:01:07.654982 18892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03: Neither blocks nor log segments found. Creating new log.
I20260926 15:01:07.655277 18892 log.cc:824] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03: Log is configured to *not* fsync() on all Append() calls
I20260926 15:01:07.656111 18892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03: No bootstrap required, opened a new log
I20260926 15:01:07.657778 18892 raft_consensus.cc:348] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02c290eb0a543b19da8aabe77102c03" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 39521 } }
I20260926 15:01:07.657968 18892 raft_consensus.cc:374] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:01:07.658020 18892 raft_consensus.cc:729] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d02c290eb0a543b19da8aabe77102c03, State: Initialized, Role: FOLLOWER
I20260926 15:01:07.658197 18892 consensus_queue.cc:260] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02c290eb0a543b19da8aabe77102c03" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 39521 } }
I20260926 15:01:07.658313 18892 raft_consensus.cc:388] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:01:07.658355 18892 raft_consensus.cc:482] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:01:07.658407 18892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:01:07.658493 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 18824
I20260926 15:01:07.658598 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:01:07.658825 8600 external_mini_cluster.cc:1428] Setting key abc77cc4bb9c434357ffcdd491430e46
I20260926 15:01:07.659657 18892 raft_consensus.cc:504] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02c290eb0a543b19da8aabe77102c03" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 39521 } }
I20260926 15:01:07.659754 18892 leader_election.cc:302] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d02c290eb0a543b19da8aabe77102c03; no voters:
I20260926 15:01:07.659973 18892 leader_election.cc:288] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:01:07.660024 18897 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:01:07.660277 18892 sys_catalog.cc:564] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:01:07.660310 18897 raft_consensus.cc:686] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03 [term 1 LEADER]: Becoming Leader. State: Replica: d02c290eb0a543b19da8aabe77102c03, State: Running, Role: LEADER
I20260926 15:01:07.660490 18897 consensus_queue.cc:237] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02c290eb0a543b19da8aabe77102c03" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 39521 } }
I20260926 15:01:07.660975 18898 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d02c290eb0a543b19da8aabe77102c03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02c290eb0a543b19da8aabe77102c03" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 39521 } } }
I20260926 15:01:07.661068 18898 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03 [sys.catalog]: This master's current role is: LEADER
I20260926 15:01:07.661214 18899 sys_catalog.cc:455] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d02c290eb0a543b19da8aabe77102c03. Latest consensus state: current_term: 1 leader_uuid: "d02c290eb0a543b19da8aabe77102c03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d02c290eb0a543b19da8aabe77102c03" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 39521 } } }
I20260926 15:01:07.661294 18899 sys_catalog.cc:458] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03 [sys.catalog]: This master's current role is: LEADER
I20260926 15:01:07.661442 18906 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:01:07.661963 18906 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:01:07.663789 18906 catalog_manager.cc:1347] Generated new cluster ID: 9a81975a0996470791f38008f856fcd6
I20260926 15:01:07.663858 18906 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 15:01:07 dist-test-slave-jnhs krb5kdc[18811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:01:07.672626 18906 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:01:07.673193 18906 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:01:07.675555 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:07.659382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52534 (local address 127.8.102.62:39521)
0926 15:01:07.659584 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:07.659596 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:07.659639 (+ 43us) server_negotiation.cc:407] Connection header received
0926 15:01:07.659781 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:07.659789 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:07.659868 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:07.660044 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:07.661006 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:07.662753 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:07.666875 (+ 4122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:07.667648 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:07.667827 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:07.671039 (+ 3212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:01:07.671045 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:01:07.671045 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:01:07.671061 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:01:07.673471 (+ 2410us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:01:07.673956 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:01:07.673961 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:01:07.673962 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:01:07.674112 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:01:07.674393 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:01:07.674395 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:01:07.674395 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:01:07.674826 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:01:07.674998 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:07.675105 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 15:01:07.675276 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":101,"threads_started":1}
I20260926 15:01:07.679557 18906 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03: Generated new TSK 0
I20260926 15:01:07.679780 18906 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
I20260926 15:01:07.711861 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:39521
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:01:07.811836 18918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:01:07.812088 18918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:01:07.812117 18918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:01:07.813493 18918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:01:07.813551 18918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:01:07.813583 18918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:01:07.815295 18918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:39521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:01:07.815590 18918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:01:07.815866 18918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:01:07.821480 18923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:07.821483 18926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:07.821516 18918 server_base.cc:1034] running on GCE node
W20260926 15:01:07.821483 18924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:07.822019 18918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:01:07.822429 18918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:01:07.823628 18918 hybrid_clock.cc:648] HybridClock initialized: now 1790434867823596 us; error 49 us; skew 500 ppm
Sep 26 15:01:07 dist-test-slave-jnhs krb5kdc[18811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434867, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:01:07.826467 18918 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:01:07.827205 18918 webserver.cc:460] Webserver started at http://127.8.102.1:36745/ using document root <none> and password file <none>
I20260926 15:01:07.827498 18918 fs_manager.cc:359] Metadata directory not provided
I20260926 15:01:07.827579 18918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:01:07.827734 18918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:01:07.828971 18918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "26872bb81f0a4e37b4a58d8a580a21e3"
format_stamp: "Formatted at 2026-09-26 15:01:07 on dist-test-slave-jnhs"
server_key: "7cc780ee8067c5795507ac4d1df0d29a"
server_key_iv: "c1fd40656e4507945253e35220844eba"
server_key_version: "encryptionkey@0"
I20260926 15:01:07.829435 18918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "26872bb81f0a4e37b4a58d8a580a21e3"
format_stamp: "Formatted at 2026-09-26 15:01:07 on dist-test-slave-jnhs"
server_key: "7cc780ee8067c5795507ac4d1df0d29a"
server_key_iv: "c1fd40656e4507945253e35220844eba"
server_key_version: "encryptionkey@0"
I20260926 15:01:07.831135 18918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.004s
I20260926 15:01:07.832044 18933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:07.832403 18918 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:01:07.832551 18918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "26872bb81f0a4e37b4a58d8a580a21e3"
format_stamp: "Formatted at 2026-09-26 15:01:07 on dist-test-slave-jnhs"
server_key: "7cc780ee8067c5795507ac4d1df0d29a"
server_key_iv: "c1fd40656e4507945253e35220844eba"
server_key_version: "encryptionkey@0"
I20260926 15:01:07.832641 18918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:01:07.854898 18918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:01:07.857770 18918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:01:07.857985 18918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:01:07.858275 18918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:01:07.858621 18918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:01:07.858736 18918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:07.858780 18918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:01:07.858825 18918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:07.865284 18918 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:35629
I20260926 15:01:07.865387 19046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:35629 every 8 connection(s)
I20260926 15:01:07.865746 18918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:01:07.867791 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 18918
I20260926 15:01:07.867931 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:01:07.868170 8600 external_mini_cluster.cc:1428] Setting key 56edaac4aa4def537f2d866737daf8b0
Sep 26 15:01:07 dist-test-slave-jnhs krb5kdc[18811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434867, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.102.2@KRBTEST.COM; defaulting to no policy
I20260926 15:01:07.878369 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:07.866752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:34079 (local address 127.8.102.62:39521)
0926 15:01:07.866818 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:07.866820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:07.867066 (+ 246us) server_negotiation.cc:407] Connection header received
0926 15:01:07.867603 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:07.867605 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:07.867616 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:07.867718 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:07.869151 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:07.869987 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:07.871577 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:07.871917 (+ 340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:07.872004 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:07.874481 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:01:07.874483 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:01:07.874483 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:01:07.874494 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:01:07.876501 (+ 2007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:01:07.877027 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:01:07.877029 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:01:07.877030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:01:07.877095 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:01:07.877422 (+ 327us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:01:07.877423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:01:07.877423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:01:07.877604 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:01:07.877677 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:07.878213 (+ 536us) server_negotiation.cc:299] Negotiation successful
0926 15:01:07.878298 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 15:01:07.878460 19049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:07.866758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:39521 (local address 127.8.102.1:34079)
0926 15:01:07.866918 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:07.866962 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:07.867473 (+ 511us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:07.867818 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:07.867822 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:07.868700 (+ 878us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:07.869018 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:07.869026 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:07.870108 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:07.870109 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:07.871463 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:07.871468 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:07.872029 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:07.872030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:07.872152 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:07.872865 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:07.872877 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:07.874360 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:01:07.876612 (+ 2252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:01:07.876622 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:01:07.876624 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:01:07.876931 (+ 307us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:01:07.877196 (+ 265us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:01:07.877197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:01:07.877197 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:01:07.877326 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:01:07.877699 (+ 373us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:01:07.877706 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:01:07.877971 (+ 265us) client_negotiation.cc:769] Sending connection context
0926 15:01:07.878095 (+ 124us) client_negotiation.cc:240] Negotiation successful
0926 15:01:07.878248 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":85,"threads_started":1}
I20260926 15:01:07.879280 19047 heartbeater.cc:347] Connected to a master server at 127.8.102.62:39521
I20260926 15:01:07.879383 19047 heartbeater.cc:464] Registering TS with master...
I20260926 15:01:07.879581 19047 heartbeater.cc:511] Master 127.8.102.62:39521 requested a full tablet report, sending...
I20260926 15:01:07.880257 18856 ts_manager.cc:194] Registered new tserver with Master: 26872bb81f0a4e37b4a58d8a580a21e3 (127.8.102.1:35629)
I20260926 15:01:07.880895 18856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:34079
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
I20260926 15:01:07.901643 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:39521
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:01:07.987224 19052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:01:07.987458 19052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:01:07.987485 19052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:01:07.988785 19052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:01:07.988847 19052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:01:07.988879 19052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:01:07.990463 19052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:39521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19052
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:01:07.990770 19052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:01:07.991042 19052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:01:07.996379 19060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:07.996393 19057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:07.996376 19058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:07.996439 19052 server_base.cc:1034] running on GCE node
I20260926 15:01:07.996783 19052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:01:07.997126 19052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:01:07.998293 19052 hybrid_clock.cc:648] HybridClock initialized: now 1790434867998283 us; error 35 us; skew 500 ppm
Sep 26 15:01:08 dist-test-slave-jnhs krb5kdc[18811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434868, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:01:08.006315 19052 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:01:08.007048 19052 webserver.cc:460] Webserver started at http://127.8.102.2:33291/ using document root <none> and password file <none>
I20260926 15:01:08.007344 19052 fs_manager.cc:359] Metadata directory not provided
I20260926 15:01:08.007421 19052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:01:08.007577 19052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:01:08.008839 19052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "e25133c7394a40c7a7d96e67535e80b2"
format_stamp: "Formatted at 2026-09-26 15:01:08 on dist-test-slave-jnhs"
server_key: "19eb241eb706ddf1f3a27d08cd365e84"
server_key_iv: "ef38d8dccba1bc47544429bbc37d8348"
server_key_version: "encryptionkey@0"
I20260926 15:01:08.009269 19052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "e25133c7394a40c7a7d96e67535e80b2"
format_stamp: "Formatted at 2026-09-26 15:01:08 on dist-test-slave-jnhs"
server_key: "19eb241eb706ddf1f3a27d08cd365e84"
server_key_iv: "ef38d8dccba1bc47544429bbc37d8348"
server_key_version: "encryptionkey@0"
I20260926 15:01:08.010970 19052 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 15:01:08.011991 19067 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:08.012239 19052 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:01:08.012342 19052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "e25133c7394a40c7a7d96e67535e80b2"
format_stamp: "Formatted at 2026-09-26 15:01:08 on dist-test-slave-jnhs"
server_key: "19eb241eb706ddf1f3a27d08cd365e84"
server_key_iv: "ef38d8dccba1bc47544429bbc37d8348"
server_key_version: "encryptionkey@0"
I20260926 15:01:08.012406 19052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:01:08.034250 19052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:01:08.037225 19052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:01:08.037431 19052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:01:08.037735 19052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:01:08.038136 19052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:01:08.038193 19052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:08.038236 19052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:01:08.038265 19052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:08.044890 19052 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:34421
I20260926 15:01:08.044970 19180 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:34421 every 8 connection(s)
I20260926 15:01:08.045334 19052 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:01:08.047638 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 19052
I20260926 15:01:08.047767 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:01:08.047982 8600 external_mini_cluster.cc:1428] Setting key 33c10e349d2cf7dbd9885722e71c74ae
Sep 26 15:01:08 dist-test-slave-jnhs krb5kdc[18811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434868, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.102.3@KRBTEST.COM; defaulting to no policy
I20260926 15:01:08.057180 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:08.046256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:45085 (local address 127.8.102.62:39521)
0926 15:01:08.046305 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:08.046307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:08.046493 (+ 186us) server_negotiation.cc:407] Connection header received
0926 15:01:08.047099 (+ 606us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:08.047100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:08.047113 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:08.047187 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:08.048489 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.049382 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.050798 (+ 1416us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.051004 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.051082 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:08.053896 (+ 2814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:01:08.053897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:01:08.053898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:01:08.053909 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:01:08.055503 (+ 1594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:01:08.056004 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:01:08.056006 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:01:08.056006 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:01:08.056067 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:01:08.056341 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:01:08.056342 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:01:08.056342 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:01:08.056466 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:01:08.056545 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:08.057004 (+ 459us) server_negotiation.cc:299] Negotiation successful
0926 15:01:08.057086 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 15:01:08.057276 19183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:08.046168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:39521 (local address 127.8.102.2:45085)
0926 15:01:08.046332 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:08.046376 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:08.046965 (+ 589us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:08.047262 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:08.047266 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:08.047969 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:08.048348 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:08.048373 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.049493 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.049494 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:08.050608 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:08.050610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.051102 (+ 492us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.051103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:08.051280 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:08.052025 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:08.052035 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:08.053777 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:01:08.055620 (+ 1843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:01:08.055627 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:01:08.055630 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:01:08.055913 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:01:08.056140 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:01:08.056140 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:01:08.056141 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:01:08.056260 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:01:08.056536 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:01:08.056542 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:01:08.056697 (+ 155us) client_negotiation.cc:769] Sending connection context
0926 15:01:08.056823 (+ 126us) client_negotiation.cc:240] Negotiation successful
0926 15:01:08.057023 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":89,"threads_started":1}
I20260926 15:01:08.057936 19181 heartbeater.cc:347] Connected to a master server at 127.8.102.62:39521
I20260926 15:01:08.058032 19181 heartbeater.cc:464] Registering TS with master...
I20260926 15:01:08.058185 19181 heartbeater.cc:511] Master 127.8.102.62:39521 requested a full tablet report, sending...
I20260926 15:01:08.058609 18856 ts_manager.cc:194] Registered new tserver with Master: e25133c7394a40c7a7d96e67535e80b2 (127.8.102.2:34421)
I20260926 15:01:08.059141 18856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:45085
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
I20260926 15:01:08.079008 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:39521
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:01:08.165666 19186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:01:08.165897 19186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:01:08.165925 19186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:01:08.167204 19186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:01:08.167254 19186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:01:08.167284 19186 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:01:08.168805 19186 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:39521
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19186
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:01:08.169083 19186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:01:08.169337 19186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:01:08.174882 19192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:08.174929 19194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:08.174938 19191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:08.175097 19186 server_base.cc:1034] running on GCE node
I20260926 15:01:08.175437 19186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:01:08.175715 19186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:01:08.176905 19186 hybrid_clock.cc:648] HybridClock initialized: now 1790434868176883 us; error 40 us; skew 500 ppm
Sep 26 15:01:08 dist-test-slave-jnhs krb5kdc[18811](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434868, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:01:08.179873 19186 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:01:08.180413 19186 webserver.cc:460] Webserver started at http://127.8.102.3:38425/ using document root <none> and password file <none>
I20260926 15:01:08.180657 19186 fs_manager.cc:359] Metadata directory not provided
I20260926 15:01:08.180729 19186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:01:08.180908 19186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:01:08.182039 19186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "a4dd45feda984dc9b20d8ffd3c3ac5af"
format_stamp: "Formatted at 2026-09-26 15:01:08 on dist-test-slave-jnhs"
server_key: "1ace2441bee359506324db7be8bb5e59"
server_key_iv: "2dd410482bbb213d4d8d5b9c867b5a20"
server_key_version: "encryptionkey@0"
I20260926 15:01:08.182426 19186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "a4dd45feda984dc9b20d8ffd3c3ac5af"
format_stamp: "Formatted at 2026-09-26 15:01:08 on dist-test-slave-jnhs"
server_key: "1ace2441bee359506324db7be8bb5e59"
server_key_iv: "2dd410482bbb213d4d8d5b9c867b5a20"
server_key_version: "encryptionkey@0"
I20260926 15:01:08.183902 19186 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:01:08.184931 19201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:08.185189 19186 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:01:08.185359 19186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "a4dd45feda984dc9b20d8ffd3c3ac5af"
format_stamp: "Formatted at 2026-09-26 15:01:08 on dist-test-slave-jnhs"
server_key: "1ace2441bee359506324db7be8bb5e59"
server_key_iv: "2dd410482bbb213d4d8d5b9c867b5a20"
server_key_version: "encryptionkey@0"
I20260926 15:01:08.185449 19186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:01:08.197906 19186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:01:08.200654 19186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:01:08.200884 19186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:01:08.201159 19186 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:01:08.201519 19186 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:01:08.201579 19186 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:08.201624 19186 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:01:08.201653 19186 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:08.207883 19186 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:35777
I20260926 15:01:08.207965 19314 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:35777 every 8 connection(s)
I20260926 15:01:08.208338 19186 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:01:08.214915 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 19186
I20260926 15:01:08.215049 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:01:08.215246 8600 external_mini_cluster.cc:1428] Setting key 30e40e6b94c9737a490ef151c2917473
Sep 26 15:01:08 dist-test-slave-jnhs krb5kdc[18811](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434868, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:01:08.219604 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:08.209277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:47715 (local address 127.8.102.62:39521)
0926 15:01:08.209407 (+ 130us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:08.209410 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:08.209556 (+ 146us) server_negotiation.cc:407] Connection header received
0926 15:01:08.210054 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:08.210056 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:08.210068 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:08.210143 (+ 75us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:08.211307 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.211857 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.212928 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.213133 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.213209 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:08.216044 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:01:08.216046 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:01:08.216046 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:01:08.216056 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:01:08.217894 (+ 1838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:01:08.218457 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:01:08.218458 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:01:08.218459 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:01:08.218528 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:01:08.218875 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:01:08.218876 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:01:08.218877 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:01:08.219007 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:01:08.219091 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:08.219408 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 15:01:08.219503 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 15:01:08.219828 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:08.209164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:39521 (local address 127.8.102.3:47715)
0926 15:01:08.209348 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:08.209433 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:08.209948 (+ 515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:08.210225 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:08.210229 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:08.210954 (+ 725us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:08.211170 (+ 216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:08.211176 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.211964 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.211965 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:08.212829 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:08.212832 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.213240 (+ 408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.213241 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:08.213372 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:08.214069 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:08.214079 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:08.215879 (+ 1800us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:01:08.218048 (+ 2169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:01:08.218059 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:01:08.218062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:01:08.218364 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:01:08.218611 (+ 247us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:01:08.218611 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:01:08.218612 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:01:08.218800 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:01:08.219081 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:01:08.219088 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:01:08.219258 (+ 170us) client_negotiation.cc:769] Sending connection context
0926 15:01:08.219375 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 15:01:08.219559 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":99,"threads_started":1}
I20260926 15:01:08.220405 19315 heartbeater.cc:347] Connected to a master server at 127.8.102.62:39521
I20260926 15:01:08.220561 19315 heartbeater.cc:464] Registering TS with master...
I20260926 15:01:08.220839 19315 heartbeater.cc:511] Master 127.8.102.62:39521 requested a full tablet report, sending...
I20260926 15:01:08.221300 18856 ts_manager.cc:194] Registered new tserver with Master: a4dd45feda984dc9b20d8ffd3c3ac5af (127.8.102.3:35777)
I20260926 15:01:08.221664 18856 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:47715
I20260926 15:01:08.227968 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:01:08.237283 18896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:08.229585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52548 (local address 127.8.102.62:39521)
0926 15:01:08.229685 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:08.229688 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:08.229797 (+ 109us) server_negotiation.cc:407] Connection header received
0926 15:01:08.229874 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:08.229876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:08.229887 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:08.229945 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:08.230686 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.231198 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.231982 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.232180 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.232275 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:08.233972 (+ 1697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:01:08.233974 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:01:08.233975 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:01:08.233985 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:01:08.235809 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:01:08.236215 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:01:08.236216 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:01:08.236217 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:01:08.236292 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:01:08.236647 (+ 355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:01:08.236649 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:01:08.236649 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:01:08.236853 (+ 204us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:01:08.237013 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:08.237130 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 15:01:08.237210 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 15:01:08.239642 18856 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52548:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:01:08.241338 18856 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:01:08.249908 19326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:08.244589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:35777 (local address 127.0.0.1:57022)
0926 15:01:08.245699 (+ 1110us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:08.245708 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:08.245746 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:08.246213 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:08.246214 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:08.246216 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:01:08.246263 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:08.246264 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.247749 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.247751 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:08.248780 (+ 1029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:08.248784 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.249572 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.249574 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:08.249657 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:08.249674 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 15:01:08.249750 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 15:01:08.249804 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1101,"thread_start_us":69,"threads_started":1}
I20260926 15:01:08.250788 19327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:08.244585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:34421 (local address 127.0.0.1:40892)
0926 15:01:08.245569 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:08.245579 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:08.245630 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:08.245918 (+ 288us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:08.245919 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:08.245922 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:01:08.245988 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:08.245990 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.247686 (+ 1696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.247687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:08.249254 (+ 1567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:08.249258 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.250482 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.250483 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:08.250549 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:08.250554 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:01:08.250619 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 15:01:08.250690 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"mutex_wait_us":205,"spinlock_wait_cycles":896,"thread_start_us":100,"threads_started":1}
I20260926 15:01:08.250787 19330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:08.244860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40892 (local address 127.8.102.2:34421)
0926 15:01:08.245205 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:08.245208 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:08.245710 (+ 502us) server_negotiation.cc:407] Connection header received
0926 15:01:08.245758 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:08.245761 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:08.245778 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:08.245917 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:01:08.246078 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.247559 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.249382 (+ 1823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.250282 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.250487 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:08.250564 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:08.250660 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 15:01:08.250702 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":119,"threads_started":1}
I20260926 15:01:08.250809 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:08.244678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57022 (local address 127.8.102.3:35777)
0926 15:01:08.244872 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:08.244878 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:08.245807 (+ 929us) server_negotiation.cc:407] Connection header received
0926 15:01:08.246028 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:08.246031 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:08.246050 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:08.246129 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:01:08.246442 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.247307 (+ 865us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.248892 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.249479 (+ 587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.249565 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:08.249631 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:08.249780 (+ 149us) server_negotiation.cc:299] Negotiation successful
0926 15:01:08.249824 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":156,"thread_start_us":120,"threads_started":1}
I20260926 15:01:08.250999 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:08.244585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:35629 (local address 127.0.0.1:44634)
0926 15:01:08.245299 (+ 714us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:08.245348 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:08.245432 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:08.245688 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:08.245690 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:08.245693 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:01:08.245798 (+ 105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:08.245800 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.247682 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.247683 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:08.249270 (+ 1587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:08.249272 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.250708 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.250709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:08.250791 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:08.250795 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:01:08.250849 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 15:01:08.250899 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":553,"mutex_wait_us":5,"thread_start_us":105,"threads_started":1}
I20260926 15:01:08.251636 19249 tablet_service.cc:1467] Processing CreateTablet for tablet 2c55b5c6aae44d298c9d472ddd86163e (DEFAULT_TABLE table=test-table [id=0e74c6ff95934951b4fc13bec87d0457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:01:08.251479 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:08.244861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44634 (local address 127.8.102.1:35629)
0926 15:01:08.245110 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:08.245115 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:08.245465 (+ 350us) server_negotiation.cc:407] Connection header received
0926 15:01:08.245524 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:08.245527 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:08.245547 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:08.245686 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:01:08.246080 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.247553 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.249359 (+ 1806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.250511 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.250579 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:08.250673 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:08.251368 (+ 695us) server_negotiation.cc:299] Negotiation successful
0926 15:01:08.251412 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":183,"thread_start_us":112,"threads_started":1}
I20260926 15:01:08.252010 19249 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c55b5c6aae44d298c9d472ddd86163e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:01:08.252537 18981 tablet_service.cc:1467] Processing CreateTablet for tablet 2c55b5c6aae44d298c9d472ddd86163e (DEFAULT_TABLE table=test-table [id=0e74c6ff95934951b4fc13bec87d0457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:01:08.252890 18981 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c55b5c6aae44d298c9d472ddd86163e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:01:08.256088 19332 tablet_bootstrap.cc:492] T 2c55b5c6aae44d298c9d472ddd86163e P 26872bb81f0a4e37b4a58d8a580a21e3: Bootstrap starting.
I20260926 15:01:08.256704 19333 tablet_bootstrap.cc:492] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af: Bootstrap starting.
I20260926 15:01:08.256855 19332 tablet_bootstrap.cc:654] T 2c55b5c6aae44d298c9d472ddd86163e P 26872bb81f0a4e37b4a58d8a580a21e3: Neither blocks nor log segments found. Creating new log.
I20260926 15:01:08.257463 19332 log.cc:824] T 2c55b5c6aae44d298c9d472ddd86163e P 26872bb81f0a4e37b4a58d8a580a21e3: Log is configured to *not* fsync() on all Append() calls
I20260926 15:01:08.257589 19115 tablet_service.cc:1467] Processing CreateTablet for tablet 2c55b5c6aae44d298c9d472ddd86163e (DEFAULT_TABLE table=test-table [id=0e74c6ff95934951b4fc13bec87d0457]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:01:08.257936 19115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2c55b5c6aae44d298c9d472ddd86163e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:01:08.258450 19333 tablet_bootstrap.cc:654] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af: Neither blocks nor log segments found. Creating new log.
I20260926 15:01:08.259087 19333 log.cc:824] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af: Log is configured to *not* fsync() on all Append() calls
I20260926 15:01:08.259372 19332 tablet_bootstrap.cc:492] T 2c55b5c6aae44d298c9d472ddd86163e P 26872bb81f0a4e37b4a58d8a580a21e3: No bootstrap required, opened a new log
I20260926 15:01:08.259459 19332 ts_tablet_manager.cc:1397] T 2c55b5c6aae44d298c9d472ddd86163e P 26872bb81f0a4e37b4a58d8a580a21e3: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 15:01:08.259919 19333 tablet_bootstrap.cc:492] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af: No bootstrap required, opened a new log
I20260926 15:01:08.260025 19333 ts_tablet_manager.cc:1397] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 15:01:08.261257 19336 tablet_bootstrap.cc:492] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2: Bootstrap starting.
I20260926 15:01:08.262141 19336 tablet_bootstrap.cc:654] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2: Neither blocks nor log segments found. Creating new log.
I20260926 15:01:08.262349 19332 raft_consensus.cc:348] T 2c55b5c6aae44d298c9d472ddd86163e P 26872bb81f0a4e37b4a58d8a580a21e3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25133c7394a40c7a7d96e67535e80b2" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 34421 } } peers { permanent_uuid: "a4dd45feda984dc9b20d8ffd3c3ac5af" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 35777 } } peers { permanent_uuid: "26872bb81f0a4e37b4a58d8a580a21e3" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35629 } }
I20260926 15:01:08.262816 19332 raft_consensus.cc:374] T 2c55b5c6aae44d298c9d472ddd86163e P 26872bb81f0a4e37b4a58d8a580a21e3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:01:08.262794 19333 raft_consensus.cc:348] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25133c7394a40c7a7d96e67535e80b2" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 34421 } } peers { permanent_uuid: "a4dd45feda984dc9b20d8ffd3c3ac5af" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 35777 } } peers { permanent_uuid: "26872bb81f0a4e37b4a58d8a580a21e3" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35629 } }
I20260926 15:01:08.262929 19332 raft_consensus.cc:729] T 2c55b5c6aae44d298c9d472ddd86163e P 26872bb81f0a4e37b4a58d8a580a21e3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26872bb81f0a4e37b4a58d8a580a21e3, State: Initialized, Role: FOLLOWER
I20260926 15:01:08.262974 19333 raft_consensus.cc:374] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:01:08.263017 19333 raft_consensus.cc:729] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4dd45feda984dc9b20d8ffd3c3ac5af, State: Initialized, Role: FOLLOWER
I20260926 15:01:08.263101 19336 log.cc:824] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2: Log is configured to *not* fsync() on all Append() calls
I20260926 15:01:08.263262 19332 consensus_queue.cc:260] T 2c55b5c6aae44d298c9d472ddd86163e P 26872bb81f0a4e37b4a58d8a580a21e3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25133c7394a40c7a7d96e67535e80b2" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 34421 } } peers { permanent_uuid: "a4dd45feda984dc9b20d8ffd3c3ac5af" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 35777 } } peers { permanent_uuid: "26872bb81f0a4e37b4a58d8a580a21e3" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35629 } }
I20260926 15:01:08.263206 19333 consensus_queue.cc:260] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25133c7394a40c7a7d96e67535e80b2" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 34421 } } peers { permanent_uuid: "a4dd45feda984dc9b20d8ffd3c3ac5af" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 35777 } } peers { permanent_uuid: "26872bb81f0a4e37b4a58d8a580a21e3" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35629 } }
I20260926 15:01:08.263644 19332 ts_tablet_manager.cc:1428] T 2c55b5c6aae44d298c9d472ddd86163e P 26872bb81f0a4e37b4a58d8a580a21e3: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 15:01:08.263743 19315 heartbeater.cc:503] Master 127.8.102.62:39521 was elected leader, sending a full tablet report...
I20260926 15:01:08.263792 19047 heartbeater.cc:503] Master 127.8.102.62:39521 was elected leader, sending a full tablet report...
I20260926 15:01:08.263731 19333 ts_tablet_manager.cc:1428] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 15:01:08.264480 19336 tablet_bootstrap.cc:492] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2: No bootstrap required, opened a new log
I20260926 15:01:08.264575 19336 ts_tablet_manager.cc:1397] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 15:01:08.266213 19336 raft_consensus.cc:348] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25133c7394a40c7a7d96e67535e80b2" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 34421 } } peers { permanent_uuid: "a4dd45feda984dc9b20d8ffd3c3ac5af" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 35777 } } peers { permanent_uuid: "26872bb81f0a4e37b4a58d8a580a21e3" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35629 } }
I20260926 15:01:08.266361 19336 raft_consensus.cc:374] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:01:08.266384 19336 raft_consensus.cc:729] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e25133c7394a40c7a7d96e67535e80b2, State: Initialized, Role: FOLLOWER
I20260926 15:01:08.266487 19336 consensus_queue.cc:260] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25133c7394a40c7a7d96e67535e80b2" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 34421 } } peers { permanent_uuid: "a4dd45feda984dc9b20d8ffd3c3ac5af" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 35777 } } peers { permanent_uuid: "26872bb81f0a4e37b4a58d8a580a21e3" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35629 } }
I20260926 15:01:08.266816 19336 ts_tablet_manager.cc:1428] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 15:01:08.267010 19181 heartbeater.cc:503] Master 127.8.102.62:39521 was elected leader, sending a full tablet report...
W20260926 15:01:08.295982 19182 tablet.cc:2369] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:01:08.366554 19048 tablet.cc:2369] T 2c55b5c6aae44d298c9d472ddd86163e P 26872bb81f0a4e37b4a58d8a580a21e3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:01:08.406138 19339 raft_consensus.cc:482] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:01:08.406292 19339 raft_consensus.cc:504] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25133c7394a40c7a7d96e67535e80b2" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 34421 } } peers { permanent_uuid: "a4dd45feda984dc9b20d8ffd3c3ac5af" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 35777 } } peers { permanent_uuid: "26872bb81f0a4e37b4a58d8a580a21e3" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35629 } }
I20260926 15:01:08.406889 19339 leader_election.cc:288] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e25133c7394a40c7a7d96e67535e80b2 (127.8.102.2:34421), 26872bb81f0a4e37b4a58d8a580a21e3 (127.8.102.1:35629)
I20260926 15:01:08.410022 19330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:08.407248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:51653 (local address 127.8.102.2:34421)
0926 15:01:08.407349 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:08.407351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:08.407358 (+ 7us) server_negotiation.cc:407] Connection header received
0926 15:01:08.407402 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:08.407403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:08.407415 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:08.407472 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:01:08.407732 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.408221 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.409253 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.409639 (+ 386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.409725 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:08.409736 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:08.409910 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 15:01:08.409949 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 15:01:08.410029 19317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:08.407136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:34421 (local address 127.8.102.3:51653)
0926 15:01:08.407184 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:08.407198 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:08.407250 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:08.407590 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:08.407591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:08.407593 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:01:08.407654 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:08.407656 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.408318 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.408319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:08.409139 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:08.409142 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.409724 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.409725 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:08.409812 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:08.409815 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:01:08.409896 (+ 81us) client_negotiation.cc:240] Negotiation successful
0926 15:01:08.409944 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24,"mutex_wait_us":48}
I20260926 15:01:08.410013 19331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:08.407202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:34501 (local address 127.8.102.1:35629)
0926 15:01:08.407264 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:08.407267 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:08.407422 (+ 155us) server_negotiation.cc:407] Connection header received
0926 15:01:08.407473 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:08.407474 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:08.407489 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:08.407550 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:01:08.407752 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.408222 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.409258 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.409636 (+ 378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.409723 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:08.409734 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:08.409891 (+ 157us) server_negotiation.cc:299] Negotiation successful
0926 15:01:08.409934 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 15:01:08.410120 19341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:08.407144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:35629 (local address 127.8.102.3:34501)
0926 15:01:08.407335 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:08.407345 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:08.407422 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:08.407583 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:08.407584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:08.407586 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:01:08.407665 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:08.407667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.408315 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.408316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:08.409163 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:08.409165 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.409725 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.409726 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:08.409803 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:08.409807 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:01:08.409883 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 15:01:08.409927 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"mutex_wait_us":12,"thread_start_us":95,"threads_started":1}
I20260926 15:01:08.410408 19135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c55b5c6aae44d298c9d472ddd86163e" candidate_uuid: "a4dd45feda984dc9b20d8ffd3c3ac5af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e25133c7394a40c7a7d96e67535e80b2" is_pre_election: true
I20260926 15:01:08.410439 19001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c55b5c6aae44d298c9d472ddd86163e" candidate_uuid: "a4dd45feda984dc9b20d8ffd3c3ac5af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26872bb81f0a4e37b4a58d8a580a21e3" is_pre_election: true
I20260926 15:01:08.410601 19001 raft_consensus.cc:2454] T 2c55b5c6aae44d298c9d472ddd86163e P 26872bb81f0a4e37b4a58d8a580a21e3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4dd45feda984dc9b20d8ffd3c3ac5af in term 0.
I20260926 15:01:08.410606 19135 raft_consensus.cc:2454] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4dd45feda984dc9b20d8ffd3c3ac5af in term 0.
I20260926 15:01:08.410887 19204 leader_election.cc:302] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4dd45feda984dc9b20d8ffd3c3ac5af, e25133c7394a40c7a7d96e67535e80b2; no voters:
I20260926 15:01:08.411067 19339 raft_consensus.cc:2781] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:01:08.411145 19339 raft_consensus.cc:482] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:01:08.411176 19339 raft_consensus.cc:3037] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:01:08.412004 19339 raft_consensus.cc:504] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25133c7394a40c7a7d96e67535e80b2" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 34421 } } peers { permanent_uuid: "a4dd45feda984dc9b20d8ffd3c3ac5af" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 35777 } } peers { permanent_uuid: "26872bb81f0a4e37b4a58d8a580a21e3" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35629 } }
I20260926 15:01:08.412176 19339 leader_election.cc:288] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [CANDIDATE]: Term 1 election: Requested vote from peers e25133c7394a40c7a7d96e67535e80b2 (127.8.102.2:34421), 26872bb81f0a4e37b4a58d8a580a21e3 (127.8.102.1:35629)
I20260926 15:01:08.412362 19135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c55b5c6aae44d298c9d472ddd86163e" candidate_uuid: "a4dd45feda984dc9b20d8ffd3c3ac5af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e25133c7394a40c7a7d96e67535e80b2"
I20260926 15:01:08.412358 19001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2c55b5c6aae44d298c9d472ddd86163e" candidate_uuid: "a4dd45feda984dc9b20d8ffd3c3ac5af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26872bb81f0a4e37b4a58d8a580a21e3"
I20260926 15:01:08.412442 19001 raft_consensus.cc:3037] T 2c55b5c6aae44d298c9d472ddd86163e P 26872bb81f0a4e37b4a58d8a580a21e3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:01:08.412465 19135 raft_consensus.cc:3037] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:01:08.413220 19001 raft_consensus.cc:2454] T 2c55b5c6aae44d298c9d472ddd86163e P 26872bb81f0a4e37b4a58d8a580a21e3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4dd45feda984dc9b20d8ffd3c3ac5af in term 1.
I20260926 15:01:08.413230 19135 raft_consensus.cc:2454] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4dd45feda984dc9b20d8ffd3c3ac5af in term 1.
I20260926 15:01:08.413427 19204 leader_election.cc:302] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4dd45feda984dc9b20d8ffd3c3ac5af, e25133c7394a40c7a7d96e67535e80b2; no voters:
I20260926 15:01:08.413566 19339 raft_consensus.cc:2781] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:01:08.413779 19339 raft_consensus.cc:686] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [term 1 LEADER]: Becoming Leader. State: Replica: a4dd45feda984dc9b20d8ffd3c3ac5af, State: Running, Role: LEADER
I20260926 15:01:08.413951 19339 consensus_queue.cc:237] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25133c7394a40c7a7d96e67535e80b2" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 34421 } } peers { permanent_uuid: "a4dd45feda984dc9b20d8ffd3c3ac5af" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 35777 } } peers { permanent_uuid: "26872bb81f0a4e37b4a58d8a580a21e3" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35629 } }
I20260926 15:01:08.414944 18856 catalog_manager.cc:5495] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af reported cstate change: term changed from 0 to 1, leader changed from <none> to a4dd45feda984dc9b20d8ffd3c3ac5af (127.8.102.3). New cstate: current_term: 1 leader_uuid: "a4dd45feda984dc9b20d8ffd3c3ac5af" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e25133c7394a40c7a7d96e67535e80b2" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 34421 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a4dd45feda984dc9b20d8ffd3c3ac5af" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 35777 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "26872bb81f0a4e37b4a58d8a580a21e3" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35629 } health_report { overall_health: UNKNOWN } } }
I20260926 15:01:08.438165 19329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:08.434938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57034 (local address 127.8.102.3:35777)
0926 15:01:08.435058 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:08.435061 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:08.435080 (+ 19us) server_negotiation.cc:407] Connection header received
0926 15:01:08.435132 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:08.435133 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:08.435155 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:08.435242 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:01:08.435450 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.436171 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.437159 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:08.437414 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:08.437477 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:08.437729 (+ 252us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:01:08.437843 (+ 114us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:01:08.437930 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:08.437988 (+ 58us) server_negotiation.cc:299] Negotiation successful
0926 15:01:08.438009 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260926 15:01:08.440696 19001 raft_consensus.cc:1261] T 2c55b5c6aae44d298c9d472ddd86163e P 26872bb81f0a4e37b4a58d8a580a21e3 [term 1 FOLLOWER]: Refusing update from remote peer a4dd45feda984dc9b20d8ffd3c3ac5af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:01:08.441073 19135 raft_consensus.cc:1261] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2 [term 1 FOLLOWER]: Refusing update from remote peer a4dd45feda984dc9b20d8ffd3c3ac5af: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:01:08.441125 19339 consensus_queue.cc:1035] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [LEADER]: Connected to new peer: Peer: permanent_uuid: "26872bb81f0a4e37b4a58d8a580a21e3" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 35629 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:01:08.441375 19342 consensus_queue.cc:1035] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [LEADER]: Connected to new peer: Peer: permanent_uuid: "e25133c7394a40c7a7d96e67535e80b2" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 34421 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:01:08.444569 19347 mvcc.cc:205] Tried to move back new op lower bound from 7333621221130088448 to 7333621221024284672. Current Snapshot: MvccSnapshot[applied={T|T < 7333621221130088448}]
I20260926 15:01:08.446579 18855 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:01:08.446732 18855 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52548:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:01:08.447510 18855 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d02c290eb0a543b19da8aabe77102c03: Sending DeleteTablet for 3 replicas of tablet 2c55b5c6aae44d298c9d472ddd86163e
I20260926 15:01:08.447858 19115 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c55b5c6aae44d298c9d472ddd86163e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:01:08 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:40892
I20260926 15:01:08.447927 18981 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c55b5c6aae44d298c9d472ddd86163e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:01:08 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:44634
I20260926 15:01:08.448083 19249 tablet_service.cc:1514] Processing DeleteTablet for tablet 2c55b5c6aae44d298c9d472ddd86163e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:01:08 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:57022
I20260926 15:01:08.448305 19355 tablet_replica.cc:330] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2: stopping tablet replica
I20260926 15:01:08.448323 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 18918
I20260926 15:01:08.448424 19355 raft_consensus.cc:2229] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:01:08.448987 19355 raft_consensus.cc:2258] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:01:08.449460 19355 ts_tablet_manager.cc:1905] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:01:08.451975 19355 ts_tablet_manager.cc:1918] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:01:08.452069 19355 log.cc:1196] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/2c55b5c6aae44d298c9d472ddd86163e
I20260926 15:01:08.452378 19355 ts_tablet_manager.cc:1939] T 2c55b5c6aae44d298c9d472ddd86163e P e25133c7394a40c7a7d96e67535e80b2: Deleting consensus metadata
I20260926 15:01:08.452857 18842 catalog_manager.cc:4841] TS e25133c7394a40c7a7d96e67535e80b2 (127.8.102.2:34421): tablet 2c55b5c6aae44d298c9d472ddd86163e (table test-table [id=0e74c6ff95934951b4fc13bec87d0457]) successfully deleted
I20260926 15:01:08.454823 19357 tablet_replica.cc:330] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af: stopping tablet replica
I20260926 15:01:08.455020 19357 raft_consensus.cc:2229] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [term 1 LEADER]: Raft consensus shutting down.
W20260926 15:01:08.455070 18841 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.102.1:35629 (error 108)
I20260926 15:01:08.455199 19357 raft_consensus.cc:2258] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:01:08.455750 19357 ts_tablet_manager.cc:1905] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:01:08.455750 19328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:08.455607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:35629 (local address 127.0.0.1:44650)
0926 15:01:08.455666 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:08.455705 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.1:35629: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 15:01:08.455881 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 19052
W20260926 15:01:08.456058 18841 catalog_manager.cc:4584] TS 26872bb81f0a4e37b4a58d8a580a21e3 (127.8.102.1:35629): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c55b5c6aae44d298c9d472ddd86163e: Network error: Client connection negotiation failed: client connection to 127.8.102.1:35629: connect: Connection refused (error 111)
I20260926 15:01:08.458061 19357 ts_tablet_manager.cc:1918] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:01:08.458200 19357 log.cc:1196] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/2c55b5c6aae44d298c9d472ddd86163e
I20260926 15:01:08.458551 19357 ts_tablet_manager.cc:1939] T 2c55b5c6aae44d298c9d472ddd86163e P a4dd45feda984dc9b20d8ffd3c3ac5af: Deleting consensus metadata
I20260926 15:01:08.459089 18843 catalog_manager.cc:4841] TS a4dd45feda984dc9b20d8ffd3c3ac5af (127.8.102.3:35777): tablet 2c55b5c6aae44d298c9d472ddd86163e (table test-table [id=0e74c6ff95934951b4fc13bec87d0457]) successfully deleted
I20260926 15:01:08.461939 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 19186
I20260926 15:01:08.467671 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 18824
2026-09-26T15:01:08Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (2161 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:01:08 dist-test-slave-jnhs krb5kdc[19361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:01:08 dist-test-slave-jnhs krb5kdc[19361](info): set up 2 sockets
Sep 26 15:01:08 dist-test-slave-jnhs krb5kdc[19361](info): commencing operation
krb5kdc: starting...
W20260926 15:01:09.543730 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.055s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:01:09 dist-test-slave-jnhs krb5kdc[19361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434869, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:01:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:01:09Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/rangeradmin_127.8.102.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/rangeradmin_127.8.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/rangerlookup_127.8.102.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/rangerlookup_127.8.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/HTTP_127.8.102.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/HTTP_127.8.102.20@KRBTEST.COM.keytab.
I20260926 15:01:09.731001 8600 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task17_aQt/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/postgres -l logfile start
2026-09-26 15:01:12.643 UTC [19391] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 15:01:12.643 UTC [19391] LOG: listening on IPv4 address "127.8.102.20", port 35445
2026-09-26 15:01:12.649 UTC [19391] LOG: listening on Unix socket "/tmp/.s.PGSQL.35445"
2026-09-26 15:01:12.672 UTC [19394] LOG: database system was shut down at 2026-09-26 15:01:10 UTC
2026-09-26 15:01:12.677 UTC [19391] LOG: database system is ready to accept connections
I20260926 15:01:13.656126 8600 mini_postgres.cc:96] Postgres bound to 35445
2026-09-26 15:01:13.662 UTC [19402] FATAL: database "slave" does not exist
127.8.102.20:35445 - accepting connections
I20260926 15:01:13.663290 8600 mini_ranger.cc:161] Starting Ranger...
I20260926 15:01:13.680373 8600 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 15:01:14.328598 8600 mini_ranger.cc:87] Created ranger Postgres database
I20260926 15:01:14.328716 8600 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ranger-admin
2026-09-26 15:01:14,604 [I] DB FLAVOR :POSTGRES
2026-09-26 15:01:14,604 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 15:01:14,604 [I] Checking connection..
2026-09-26 15:01:14,604 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 15:01:14,855 [I] Checking connection passed.
2026-09-26 15:01:14,856 [I] --------- Verifying version history table ---------
2026-09-26 15:01:14,856 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 15:01:15,111 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 15:01:15,111 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 15:01:15,362 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 15:01:15,362 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 15:01:15,363 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 15:01:15.619 UTC [19466] WARNING: there is no transaction in progress
2026-09-26 15:01:15,625 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 15:01:15,626 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 15:01:15,875 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 15:01:15,875 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 15:01:15,875 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 15:01:16,138 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 15:01:16,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jnhs.c.gcp-upstream.internal','N') ;"
2026-09-26 15:01:16,680 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 15:01:16,927 [I] Table x_portal_user does not exist in database ranger
2026-09-26 15:01:16,927 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 15:01:17,191 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 15:01:17,192 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 15:01:17,445 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 15:01:17,692 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 15:01:17,693 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 15:01:18.867 UTC [19574] WARNING: there is no transaction in progress
2026-09-26 15:01:18.888 UTC [19574] WARNING: there is no transaction in progress
2026-09-26 15:01:18.909 UTC [19574] WARNING: there is no transaction in progress
2026-09-26 15:01:18.929 UTC [19574] WARNING: there is no transaction in progress
2026-09-26 15:01:18.951 UTC [19574] WARNING: there is no transaction in progress
2026-09-26 15:01:18.971 UTC [19574] WARNING: there is no transaction in progress
2026-09-26 15:01:19.041 UTC [19574] WARNING: there is no transaction in progress
2026-09-26 15:01:19.050 UTC [19574] WARNING: there is no transaction in progress
2026-09-26 15:01:19.070 UTC [19574] WARNING: there is no transaction in progress
2026-09-26 15:01:19.084 UTC [19574] WARNING: there is no transaction in progress
2026-09-26 15:01:19.097 UTC [19574] WARNING: there is no transaction in progress
2026-09-26 15:01:19.116 UTC [19574] WARNING: there is no transaction in progress
2026-09-26 15:01:19.129 UTC [19574] WARNING: there is no transaction in progress
2026-09-26 15:01:19.139 UTC [19574] WARNING: there is no transaction in progress
2026-09-26 15:01:19.149 UTC [19574] WARNING: there is no transaction in progress
2026-09-26 15:01:19.158 UTC [19574] WARNING: there is no transaction in progress
2026-09-26 15:01:20,047 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 15:01:20,048 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 15:01:20,314 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 15:01:20,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 15:01:20,595 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 15:01:20,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 15:01:20,860 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 15:01:20,861 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 15:01:21,117 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 15:01:21,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jnhs.c.gcp-upstream.internal';"
2026-09-26 15:01:21,660 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 15:01:21,661 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 15:01:21,927 [I] DB_PATCHES have already been applied
I20260926 15:01:21.933676 8600 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ranger-admin:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task17_aQt/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 15:01:21.933745 8600 mini_ranger.cc:193] Using host: 127.8.102.20
I20260926 15:01:21.936939 8600 mini_ranger.cc:227] Ranger admin URL: http://127.8.102.20:42449
Sep 26, 2026 3:01:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:01:22 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:01:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 3:01:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 3:01:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 3:01:22 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 3:01:22 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 15:01:23.063103 8600 mini_ranger.cc:160] Time spent starting Ranger: real 9.400s user 0.000s sys 0.010s
Sep 26, 2026 3:01:44 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 3:01:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 3:01:44 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 3:01:44 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 3:01:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 3:01:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 3:01:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 3:01:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 3:01:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 3:01:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 3:01:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 3:01:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 3:01:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 3:01:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 3:01:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 3:01:44 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 3:01:44 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 3:01:45 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 15:01:45.818092 8600 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/rangerkms_127.8.102.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/rangerkms_127.8.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.102.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.102.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.102.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/HTTP_127.8.102.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.102.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/HTTP_127.8.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 15:01:45 dist-test-slave-jnhs krb5kdc[19361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434905, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 15:01:45.914424 8600 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 15:01:45.926002 8600 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 15:01:47.190528 8600 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 15:01:47.190623 8600 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ranger-kms
I20260926 15:01:47.190676 8600 mini_ranger_kms.cc:227] Using postgres at 127.8.102.20:35445
2026-09-26 15:01:47,364 [I] DB FLAVOR :POSTGRES
2026-09-26 15:01:47,364 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 15:01:47,364 [I] Checking connection
2026-09-26 15:01:47,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 15:01:47,637 [I] connection success
2026-09-26 15:01:47,637 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 15:01:47,637 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 15:01:47,638 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 15:01:47,886 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 15:01:47,887 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 15:01:47,887 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 15:01:47,887 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:35445/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 15:01:48,174 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 15:01:48.735738 8600 mini_ranger_kms.cc:310] Created kms service
I20260926 15:01:48.958096 8600 mini_ranger_kms.cc:326] Created kudu user
I20260926 15:01:49.005451 8600 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 15:01:49 dist-test-slave-jnhs krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434905, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.102.20@KRBTEST.COM
I20260926 15:01:49.431509 8600 mini_ranger_kms.cc:384] Added ranger policy
I20260926 15:01:49.431631 8600 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ranger-kms:/tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 15:01:49.431655 8600 mini_ranger_kms.cc:240] Using host: 127.8.102.20
I20260926 15:01:49.435111 8600 mini_ranger_kms.cc:276] Ranger KMS PID: 19755
I20260926 15:01:49.435253 8600 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.102.20:48127
Sep 26, 2026 3:01:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:01:50 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:01:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 3:01:50 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 3:01:51 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 3:01:51 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260926 15:01:51.843390 8600 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.929s user 0.006s sys 0.150s
I20260926 15:01:51.843515 8600 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 15:01:51.843608 8600 mini_ranger_kms.cc:401] 127.8.102.20:48127/kms/v1/keys
Sep 26 15:01:51 dist-test-slave-jnhs krb5kdc[19361](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790434911, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.102.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 26 15:01:52 dist-test-slave-jnhs krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434911, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.102.20@KRBTEST.COM for HTTP/127.8.102.20@KRBTEST.COM
Sep 26 15:01:52 dist-test-slave-jnhs krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434911, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.102.20@KRBTEST.COM for HTTP/127.8.102.20@KRBTEST.COM
Sep 26 15:01:55 dist-test-slave-jnhs krb5kdc[19361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:01:55.314289 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:34227
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:45701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:34227
--ranger_config_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.102.20:48127/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:01:55.424327 19805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:01:55.424569 19805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:01:55.424616 19805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:01:55.425895 19805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:01:55.425946 19805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:01:55.425961 19805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:01:55.425973 19805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:01:55.425985 19805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:01:55.427677 19805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:45701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.102.20:48127/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:34227
--ranger_config_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:34227
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:01:55.427937 19805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:01:55.428196 19805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 15:01:55.434801 19805 server_base.cc:1034] running on GCE node
W20260926 15:01:55.434823 19811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:55.434895 19813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:55.434803 19810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:55.435313 19805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:01:55.435636 19805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:01:55.436836 19805 hybrid_clock.cc:648] HybridClock initialized: now 1790434915436808 us; error 41 us; skew 500 ppm
Sep 26 15:01:55 dist-test-slave-jnhs krb5kdc[19361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434915, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:01:55.439898 19805 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:01:55.440451 19805 webserver.cc:460] Webserver started at http://127.8.102.62:33567/ using document root <none> and password file <none>
I20260926 15:01:55.440701 19805 fs_manager.cc:359] Metadata directory not provided
I20260926 15:01:55.440774 19805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:01:55.440950 19805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:01:55 dist-test-slave-jnhs krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434915, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for HTTP/127.8.102.20@KRBTEST.COM
I20260926 15:01:55.504523 19805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "313f03bf22304a31bd7d481b9b7729a7"
format_stamp: "Formatted at 2026-09-26 15:01:55 on dist-test-slave-jnhs"
server_key: "e5f4c4c2819a03d68a2696e86cfcfa47"
server_key_iv: "d770109ca40a2cad255ebaee79ef088c"
server_key_version: "kuduclusterkey@0"
I20260926 15:01:55.505040 19805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "313f03bf22304a31bd7d481b9b7729a7"
format_stamp: "Formatted at 2026-09-26 15:01:55 on dist-test-slave-jnhs"
server_key: "e5f4c4c2819a03d68a2696e86cfcfa47"
server_key_iv: "d770109ca40a2cad255ebaee79ef088c"
server_key_version: "kuduclusterkey@0"
I20260926 15:01:55.507054 19805 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260926 15:01:55.519719 19820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:55.519997 19805 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 15:01:55.520124 19805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "313f03bf22304a31bd7d481b9b7729a7"
format_stamp: "Formatted at 2026-09-26 15:01:55 on dist-test-slave-jnhs"
server_key: "e5f4c4c2819a03d68a2696e86cfcfa47"
server_key_iv: "d770109ca40a2cad255ebaee79ef088c"
server_key_version: "kuduclusterkey@0"
I20260926 15:01:55.520236 19805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:01:55.530457 19805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:01:55.532634 19805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:01:55.532793 19805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:01:55.536763 19805 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:34227
I20260926 15:01:55.536867 19872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:34227 every 8 connection(s)
I20260926 15:01:55.537226 19805 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:01:55.538019 19873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:01:55.541256 19873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7: Bootstrap starting.
I20260926 15:01:55.541733 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 19805
I20260926 15:01:55.541864 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:01:55.542040 19873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7: Neither blocks nor log segments found. Creating new log.
I20260926 15:01:55.542073 8600 external_mini_cluster.cc:1428] Setting key cfdeeee8abb029fca00cbcc246d6d06d
I20260926 15:01:55.542358 19873 log.cc:824] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7: Log is configured to *not* fsync() on all Append() calls
I20260926 15:01:55.543265 19873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7: No bootstrap required, opened a new log
I20260926 15:01:55.545183 19873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313f03bf22304a31bd7d481b9b7729a7" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 34227 } }
I20260926 15:01:55.545359 19873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:01:55.545405 19873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 313f03bf22304a31bd7d481b9b7729a7, State: Initialized, Role: FOLLOWER
I20260926 15:01:55.545532 19873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313f03bf22304a31bd7d481b9b7729a7" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 34227 } }
I20260926 15:01:55.545630 19873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:01:55.545671 19873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:01:55.545712 19873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:01:55.546290 19873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313f03bf22304a31bd7d481b9b7729a7" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 34227 } }
I20260926 15:01:55.546413 19873 leader_election.cc:302] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 313f03bf22304a31bd7d481b9b7729a7; no voters:
I20260926 15:01:55.546613 19873 leader_election.cc:288] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:01:55.546664 19878 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:01:55.546891 19878 raft_consensus.cc:686] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7 [term 1 LEADER]: Becoming Leader. State: Replica: 313f03bf22304a31bd7d481b9b7729a7, State: Running, Role: LEADER
I20260926 15:01:55.547003 19873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:01:55.547008 19878 consensus_queue.cc:237] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313f03bf22304a31bd7d481b9b7729a7" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 34227 } }
I20260926 15:01:55.547466 19878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "313f03bf22304a31bd7d481b9b7729a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313f03bf22304a31bd7d481b9b7729a7" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 34227 } } }
I20260926 15:01:55.547526 19878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7 [sys.catalog]: This master's current role is: LEADER
I20260926 15:01:55.547638 19878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 313f03bf22304a31bd7d481b9b7729a7. Latest consensus state: current_term: 1 leader_uuid: "313f03bf22304a31bd7d481b9b7729a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "313f03bf22304a31bd7d481b9b7729a7" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 34227 } } }
I20260926 15:01:55.547673 19878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7 [sys.catalog]: This master's current role is: LEADER
I20260926 15:01:55.548437 19873 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 15:01:55.549368 19884 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:01:55.549878 19884 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 15:01:55 dist-test-slave-jnhs krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434915, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:01:55.558103 19884 catalog_manager.cc:1347] Generated new cluster ID: bcd0392d59c44799968504375d206c5a
I20260926 15:01:55.558180 19884 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:01:55.558976 19877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:55.542676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50104 (local address 127.8.102.62:34227)
0926 15:01:55.542989 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:55.542998 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:55.543035 (+ 37us) server_negotiation.cc:407] Connection header received
0926 15:01:55.543606 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:55.543616 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:55.543683 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:55.543845 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:55.544627 (+ 782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:55.546490 (+ 1863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:55.547925 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:55.548531 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:55.548642 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:55.553244 (+ 4602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:01:55.553293 (+ 49us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:01:55.553294 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:01:55.553405 (+ 111us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:01:55.556890 (+ 3485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:01:55.557312 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:01:55.557315 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:01:55.557316 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:01:55.557428 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:01:55.557686 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:01:55.557687 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:01:55.557688 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:01:55.558183 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:01:55.558341 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:55.558497 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 15:01:55.558719 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":265,"thread_start_us":90,"threads_started":1}
I20260926 15:01:55.589087 19884 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:01:55.589896 19884 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:01:55.593312 19884 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7: Generated new TSK 0
I20260926 15:01:55.593587 19884 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
I20260926 15:01:56.330845 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.102.20:48127/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:34227
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:45701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:01:56.421733 19918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:01:56.421977 19918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:01:56.422019 19918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:01:56.423393 19918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:01:56.423468 19918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:01:56.423516 19918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:01:56.425308 19918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:45701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.102.20:48127/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:34227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:01:56.425652 19918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:01:56.425956 19918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:01:56.431360 19926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:56.431406 19923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:56.431340 19924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:56.431540 19918 server_base.cc:1034] running on GCE node
I20260926 15:01:56.431999 19918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:01:56.432303 19918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:01:56.433490 19918 hybrid_clock.cc:648] HybridClock initialized: now 1790434916433477 us; error 28 us; skew 500 ppm
Sep 26 15:01:56 dist-test-slave-jnhs krb5kdc[19361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:01:56.436679 19918 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:01:56.437294 19918 webserver.cc:460] Webserver started at http://127.8.102.1:40973/ using document root <none> and password file <none>
I20260926 15:01:56.437559 19918 fs_manager.cc:359] Metadata directory not provided
I20260926 15:01:56.437644 19918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:01:56.437813 19918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:01:56 dist-test-slave-jnhs krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for HTTP/127.8.102.20@KRBTEST.COM
I20260926 15:01:56.454906 19918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "0acb5ebcf0a242b1a83b07d117cf9611"
format_stamp: "Formatted at 2026-09-26 15:01:56 on dist-test-slave-jnhs"
server_key: "8f13ecbde7daaf09839286c0a1d6482c"
server_key_iv: "4572de07e7354443b5151d9159cd0d4d"
server_key_version: "kuduclusterkey@0"
I20260926 15:01:56.455492 19918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "0acb5ebcf0a242b1a83b07d117cf9611"
format_stamp: "Formatted at 2026-09-26 15:01:56 on dist-test-slave-jnhs"
server_key: "8f13ecbde7daaf09839286c0a1d6482c"
server_key_iv: "4572de07e7354443b5151d9159cd0d4d"
server_key_version: "kuduclusterkey@0"
I20260926 15:01:56.457592 19918 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:01:56.478809 19933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:56.479070 19918 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:01:56.479173 19918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "0acb5ebcf0a242b1a83b07d117cf9611"
format_stamp: "Formatted at 2026-09-26 15:01:56 on dist-test-slave-jnhs"
server_key: "8f13ecbde7daaf09839286c0a1d6482c"
server_key_iv: "4572de07e7354443b5151d9159cd0d4d"
server_key_version: "kuduclusterkey@0"
I20260926 15:01:56.479298 19918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:01:56.506906 19918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:01:56.509176 19918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:01:56.509374 19918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:01:56.509636 19918 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:01:56.509979 19918 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:01:56.510035 19918 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:56.510075 19918 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:01:56.510098 19918 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:56.518258 19918 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:41923
I20260926 15:01:56.518328 20046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:41923 every 8 connection(s)
I20260926 15:01:56.518788 19918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:01:56.528764 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 19918
I20260926 15:01:56.528925 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:01:56.529148 8600 external_mini_cluster.cc:1428] Setting key a539c697cdf08523a9b8acea8bfc6206
Sep 26 15:01:56 dist-test-slave-jnhs krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:01:56.537134 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:56.523551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:52087 (local address 127.8.102.62:34227)
0926 15:01:56.523737 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:56.523740 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:56.524000 (+ 260us) server_negotiation.cc:407] Connection header received
0926 15:01:56.524873 (+ 873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:56.524874 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:56.524885 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:56.524970 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:56.525904 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:56.526802 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:56.528584 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:56.528904 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:56.528992 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:56.532667 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:01:56.532669 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:01:56.532670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:01:56.532680 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:01:56.534983 (+ 2303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:01:56.535582 (+ 599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:01:56.535584 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:01:56.535584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:01:56.535663 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:01:56.536044 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:01:56.536045 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:01:56.536046 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:01:56.536264 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:01:56.536321 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:56.536850 (+ 529us) server_negotiation.cc:299] Negotiation successful
0926 15:01:56.536969 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":120,"threads_started":1}
I20260926 15:01:56.538110 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:56.523509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:34227 (local address 127.8.102.1:52087)
0926 15:01:56.523834 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:56.523894 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:56.524637 (+ 743us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:56.525098 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:56.525103 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:56.525413 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:56.525746 (+ 333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:56.525754 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:56.526933 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:56.526934 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:56.528431 (+ 1497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:56.528435 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:56.529084 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:56.529085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:56.529249 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:56.530128 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:56.530139 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:56.532512 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:01:56.535115 (+ 2603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:01:56.535126 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:01:56.535129 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:01:56.535461 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:01:56.535745 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:01:56.535746 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:01:56.535746 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:01:56.535941 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:01:56.536384 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:01:56.536393 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:01:56.536636 (+ 243us) client_negotiation.cc:769] Sending connection context
0926 15:01:56.537575 (+ 939us) client_negotiation.cc:240] Negotiation successful
0926 15:01:56.537806 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":274,"thread_start_us":127,"threads_started":1}
I20260926 15:01:56.540410 20047 heartbeater.cc:347] Connected to a master server at 127.8.102.62:34227
I20260926 15:01:56.540522 20047 heartbeater.cc:464] Registering TS with master...
I20260926 15:01:56.540720 20047 heartbeater.cc:511] Master 127.8.102.62:34227 requested a full tablet report, sending...
I20260926 15:01:56.541428 19837 ts_manager.cc:194] Registered new tserver with Master: 0acb5ebcf0a242b1a83b07d117cf9611 (127.8.102.1:41923)
I20260926 15:01:56.542317 19837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:52087
WARNING: no policy specified for kudu/127.8.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
I20260926 15:01:56.567061 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.102.20:48127/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:34227
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:45701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:01:56.684468 20055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:01:56.684808 20055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:01:56.684850 20055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:01:56.686945 20055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:01:56.687026 20055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:01:56.687073 20055 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:01:56.689688 20055 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:45701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.102.20:48127/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:34227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:01:56.690007 20055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:01:56.690330 20055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:01:56.696942 20065 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:56.696938 20062 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:56.697259 20063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:56.698244 20055 server_base.cc:1034] running on GCE node
I20260926 15:01:56.698493 20055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:01:56.698892 20055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:01:56.700088 20055 hybrid_clock.cc:648] HybridClock initialized: now 1790434916700054 us; error 47 us; skew 500 ppm
Sep 26 15:01:56 dist-test-slave-jnhs krb5kdc[19361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:01:56.703723 20055 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:01:56.704319 20055 webserver.cc:460] Webserver started at http://127.8.102.2:33963/ using document root <none> and password file <none>
I20260926 15:01:56.704607 20055 fs_manager.cc:359] Metadata directory not provided
I20260926 15:01:56.704671 20055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:01:56.704872 20055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:01:56 dist-test-slave-jnhs krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for HTTP/127.8.102.20@KRBTEST.COM
I20260926 15:01:56.719537 20055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "c09ed35f8a914b308bedd0bbb786be69"
format_stamp: "Formatted at 2026-09-26 15:01:56 on dist-test-slave-jnhs"
server_key: "fa2e73980bdbdaea7bb3e7790bce9ca8"
server_key_iv: "7ab05fe91314b9c12fd5f0dd1593bc6d"
server_key_version: "kuduclusterkey@0"
I20260926 15:01:56.720122 20055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "c09ed35f8a914b308bedd0bbb786be69"
format_stamp: "Formatted at 2026-09-26 15:01:56 on dist-test-slave-jnhs"
server_key: "fa2e73980bdbdaea7bb3e7790bce9ca8"
server_key_iv: "7ab05fe91314b9c12fd5f0dd1593bc6d"
server_key_version: "kuduclusterkey@0"
I20260926 15:01:56.723999 20055 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.002s sys 0.000s
Sep 26 15:01:56 dist-test-slave-jnhs krb5kdc[19361](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790434916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:01:56.746657 20072 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:56.746944 20055 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.001s
I20260926 15:01:56.747040 20055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "c09ed35f8a914b308bedd0bbb786be69"
format_stamp: "Formatted at 2026-09-26 15:01:56 on dist-test-slave-jnhs"
server_key: "fa2e73980bdbdaea7bb3e7790bce9ca8"
server_key_iv: "7ab05fe91314b9c12fd5f0dd1593bc6d"
server_key_version: "kuduclusterkey@0"
I20260926 15:01:56.747136 20055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:01:56.762041 20055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:01:56.764696 20055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:01:56.764875 20055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:01:56.765089 20055 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:01:56.765470 20055 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:01:56.765517 20055 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:56.765554 20055 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:01:56.765571 20055 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:56.773875 20055 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:35093
I20260926 15:01:56.773952 20185 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:35093 every 8 connection(s)
I20260926 15:01:56.774320 20055 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
Sep 26 15:01:56 dist-test-slave-jnhs krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:01:56.783509 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 20055
I20260926 15:01:56.783631 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:01:56.783835 8600 external_mini_cluster.cc:1428] Setting key d00459b221f1f0c05199cd5321e4b682
I20260926 15:01:56.785137 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:56.775182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:47065 (local address 127.8.102.62:34227)
0926 15:01:56.775253 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:56.775255 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:56.775442 (+ 187us) server_negotiation.cc:407] Connection header received
0926 15:01:56.776019 (+ 577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:56.776020 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:56.776032 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:56.776110 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:56.776746 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:56.777547 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:56.778875 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:56.779087 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:56.779183 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:56.781660 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:01:56.781662 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:01:56.781663 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:01:56.781673 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:01:56.783230 (+ 1557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:01:56.783882 (+ 652us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:01:56.783883 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:01:56.783884 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:01:56.783948 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:01:56.784363 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:01:56.784364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:01:56.784364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:01:56.784528 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:01:56.784605 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:56.784943 (+ 338us) server_negotiation.cc:299] Negotiation successful
0926 15:01:56.785042 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 15:01:56.785416 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:56.775143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:34227 (local address 127.8.102.2:47065)
0926 15:01:56.775307 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:56.775352 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:56.775927 (+ 575us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:56.776183 (+ 256us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:56.776187 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:56.776449 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:56.776654 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:56.776659 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:56.777673 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:56.777674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:56.778772 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:56.778776 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:56.779210 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:56.779211 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:56.779373 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:56.780011 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:56.780021 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:56.781537 (+ 1516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:01:56.783454 (+ 1917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:01:56.783463 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:01:56.783466 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:01:56.783796 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:01:56.784116 (+ 320us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:01:56.784117 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:01:56.784119 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:01:56.784290 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:01:56.784608 (+ 318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:01:56.784615 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:01:56.784797 (+ 182us) client_negotiation.cc:769] Sending connection context
0926 15:01:56.784920 (+ 123us) client_negotiation.cc:240] Negotiation successful
0926 15:01:56.785083 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":84,"threads_started":1}
I20260926 15:01:56.785854 20186 heartbeater.cc:347] Connected to a master server at 127.8.102.62:34227
I20260926 15:01:56.785946 20186 heartbeater.cc:464] Registering TS with master...
I20260926 15:01:56.786144 20186 heartbeater.cc:511] Master 127.8.102.62:34227 requested a full tablet report, sending...
I20260926 15:01:56.786535 19837 ts_manager.cc:194] Registered new tserver with Master: c09ed35f8a914b308bedd0bbb786be69 (127.8.102.2:35093)
I20260926 15:01:56.787087 19837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:47065
WARNING: no policy specified for kudu/127.8.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
I20260926 15:01:56.814917 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.102.20:48127/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:34227
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:45701
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:01:56.938254 20192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:01:56.938675 20192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:01:56.938807 20192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:01:56.941025 20192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:01:56.941146 20192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:01:56.941233 20192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:01:56.943795 20192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:45701
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.102.20:48127/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:34227
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:01:56.944326 20192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:01:56.944675 20192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:01:56.951849 20200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:56.951876 20198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:01:56.952169 20197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:01:56.952531 20192 server_base.cc:1034] running on GCE node
I20260926 15:01:56.952785 20192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:01:56.953095 20192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:01:56.954281 20192 hybrid_clock.cc:648] HybridClock initialized: now 1790434916954257 us; error 35 us; skew 500 ppm
Sep 26 15:01:56 dist-test-slave-jnhs krb5kdc[19361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:01:56.957748 20192 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:01:56.958359 20192 webserver.cc:460] Webserver started at http://127.8.102.3:37551/ using document root <none> and password file <none>
I20260926 15:01:56.958673 20192 fs_manager.cc:359] Metadata directory not provided
I20260926 15:01:56.958740 20192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:01:56.958884 20192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:01:56 dist-test-slave-jnhs krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for HTTP/127.8.102.20@KRBTEST.COM
I20260926 15:01:56.971774 20192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "e06c709a62c04118a7a11d55b1d1caba"
format_stamp: "Formatted at 2026-09-26 15:01:56 on dist-test-slave-jnhs"
server_key: "f6ecfa9ab576f97ee36d8815a0102dbc"
server_key_iv: "e74c4ec47d5d445ca119dd26dadd2776"
server_key_version: "kuduclusterkey@0"
I20260926 15:01:56.972276 20192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "e06c709a62c04118a7a11d55b1d1caba"
format_stamp: "Formatted at 2026-09-26 15:01:56 on dist-test-slave-jnhs"
server_key: "f6ecfa9ab576f97ee36d8815a0102dbc"
server_key_iv: "e74c4ec47d5d445ca119dd26dadd2776"
server_key_version: "kuduclusterkey@0"
I20260926 15:01:56.975137 20192 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.003s sys 0.000s
I20260926 15:01:56.990782 20207 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:56.990963 20192 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.000s
I20260926 15:01:56.991043 20192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "e06c709a62c04118a7a11d55b1d1caba"
format_stamp: "Formatted at 2026-09-26 15:01:56 on dist-test-slave-jnhs"
server_key: "f6ecfa9ab576f97ee36d8815a0102dbc"
server_key_iv: "e74c4ec47d5d445ca119dd26dadd2776"
server_key_version: "kuduclusterkey@0"
I20260926 15:01:56.991142 20192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:01:57.017096 20192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:01:57.019743 20192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:01:57.019917 20192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:01:57.020140 20192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:01:57.020539 20192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:01:57.020591 20192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:57.020627 20192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:01:57.020650 20192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:01:57.028950 20192 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:36429
I20260926 15:01:57.029361 20192 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:01:57.029909 20320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:36429 every 8 connection(s)
I20260926 15:01:57.032850 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 20192
I20260926 15:01:57.032971 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:01:57.033157 8600 external_mini_cluster.cc:1428] Setting key dcc6d0b09f5cd354c947a23f8a3a0796
Sep 26 15:01:57 dist-test-slave-jnhs krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:01:57.041494 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:57.030184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:59777 (local address 127.8.102.62:34227)
0926 15:01:57.030243 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:57.030246 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:57.030446 (+ 200us) server_negotiation.cc:407] Connection header received
0926 15:01:57.031188 (+ 742us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:57.031190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:57.031204 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:57.031238 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:57.032297 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.033002 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.034730 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.035042 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.035126 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:57.037886 (+ 2760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:01:57.037888 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:01:57.037889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:01:57.037898 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:01:57.039700 (+ 1802us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:01:57.040252 (+ 552us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:01:57.040254 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:01:57.040254 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:01:57.040330 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:01:57.040638 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:01:57.040639 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:01:57.040639 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:01:57.040786 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:01:57.040830 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:57.041284 (+ 454us) server_negotiation.cc:299] Negotiation successful
0926 15:01:57.041388 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 15:01:57.041939 20323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:57.030174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:34227 (local address 127.8.102.3:59777)
0926 15:01:57.030320 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:57.030364 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:57.031048 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:57.031459 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:57.031464 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:57.031718 (+ 254us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:01:57.032148 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:57.032156 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.033136 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.033137 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:57.034623 (+ 1486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:57.034626 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.035164 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.035165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:57.035342 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:57.035943 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:01:57.035954 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:01:57.037749 (+ 1795us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:01:57.039808 (+ 2059us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:01:57.039817 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:01:57.039820 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:01:57.040156 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:01:57.040405 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:01:57.040406 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:01:57.040406 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:01:57.040553 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:01:57.040880 (+ 327us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:01:57.040886 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:01:57.041096 (+ 210us) client_negotiation.cc:769] Sending connection context
0926 15:01:57.041217 (+ 121us) client_negotiation.cc:240] Negotiation successful
0926 15:01:57.041404 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":80,"threads_started":1}
I20260926 15:01:57.042629 20321 heartbeater.cc:347] Connected to a master server at 127.8.102.62:34227
I20260926 15:01:57.042755 20321 heartbeater.cc:464] Registering TS with master...
I20260926 15:01:57.042954 20321 heartbeater.cc:511] Master 127.8.102.62:34227 requested a full tablet report, sending...
I20260926 15:01:57.043421 19837 ts_manager.cc:194] Registered new tserver with Master: e06c709a62c04118a7a11d55b1d1caba (127.8.102.3:36429)
I20260926 15:01:57.043902 19837 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:59777
I20260926 15:01:57.044754 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:01:57.060007 20050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:57.046996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50114 (local address 127.8.102.62:34227)
0926 15:01:57.047066 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:57.047068 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:57.050922 (+ 3854us) server_negotiation.cc:407] Connection header received
0926 15:01:57.051009 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:57.051011 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:57.051025 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:57.051097 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:01:57.051875 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.052652 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.053776 (+ 1124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.054077 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.054154 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:57.055923 (+ 1769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:01:57.055925 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:01:57.055926 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:01:57.055940 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:01:57.057797 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:01:57.059078 (+ 1281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:01:57.059080 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:01:57.059081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:01:57.059159 (+ 78us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:01:57.059390 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:01:57.059391 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:01:57.059392 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:01:57.059543 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:01:57.059649 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:57.059805 (+ 156us) server_negotiation.cc:299] Negotiation successful
0926 15:01:57.059900 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 15:01:57.061926 19837 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50114:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:01:57.064353 19837 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:01:57.073961 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:57.067844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41510 (local address 127.8.102.1:41923)
0926 15:01:57.068194 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:57.068199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:57.068837 (+ 638us) server_negotiation.cc:407] Connection header received
0926 15:01:57.069031 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:57.069034 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:57.069049 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:57.069101 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:01:57.069302 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.070736 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.072289 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.073429 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.073480 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:57.073540 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:57.073828 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 15:01:57.073875 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":73,"threads_started":1}
I20260926 15:01:57.074486 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:57.067753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:41923 (local address 127.0.0.1:41510)
0926 15:01:57.068773 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:57.068783 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:57.068942 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:57.069184 (+ 242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:57.069185 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:57.069187 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:01:57.069230 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:57.069231 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.070823 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.070823 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:57.072179 (+ 1356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:57.072183 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.073603 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.073604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:57.073702 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:57.073719 (+ 17us) client_negotiation.cc:769] Sending connection context
0926 15:01:57.074319 (+ 600us) client_negotiation.cc:240] Negotiation successful
0926 15:01:57.074402 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"thread_start_us":87,"threads_started":1}
I20260926 15:01:57.075428 19973 tablet_service.cc:1467] Processing CreateTablet for tablet 911f75a3b574491db8827b558033527d (DEFAULT_TABLE table=test-table [id=d564c86d08ac42088ac7e88c6c669a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:01:57.075587 20335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:57.067733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:35093 (local address 127.0.0.1:34480)
0926 15:01:57.067976 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:57.068016 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:57.068080 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:57.068640 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:57.068644 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:57.068646 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:01:57.068731 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:57.068733 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.071453 (+ 2720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.071455 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:57.072590 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:57.072593 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.075355 (+ 2762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.075356 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:57.075439 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:57.075443 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:01:57.075474 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 15:01:57.075515 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"thread_start_us":79,"threads_started":1}
I20260926 15:01:57.075839 19973 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911f75a3b574491db8827b558033527d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:01:57.076148 20338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:57.068253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.8.102.2:35093)
0926 15:01:57.068452 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:57.068456 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:57.068467 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:01:57.068508 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:57.068510 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:57.068526 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:57.068804 (+ 278us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:01:57.068870 (+ 66us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.070129 (+ 1259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.073148 (+ 3019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.075230 (+ 2082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.075893 (+ 663us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:57.075993 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:57.076023 (+ 30us) server_negotiation.cc:299] Negotiation successful
0926 15:01:57.076067 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":97,"threads_started":1}
I20260926 15:01:57.076800 20118 tablet_service.cc:1467] Processing CreateTablet for tablet 911f75a3b574491db8827b558033527d (DEFAULT_TABLE table=test-table [id=d564c86d08ac42088ac7e88c6c669a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:01:57.077183 20118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911f75a3b574491db8827b558033527d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:01:57.080857 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:57.067962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:36429 (local address 127.0.0.1:40646)
0926 15:01:57.075000 (+ 7038us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:57.075012 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:57.075058 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:57.075341 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:57.075342 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:57.075343 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:01:57.075378 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:57.075380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.077677 (+ 2297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.077678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:57.078766 (+ 1088us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:57.078770 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.080617 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.080618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:57.080681 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:57.080685 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:01:57.080736 (+ 51us) client_negotiation.cc:240] Negotiation successful
0926 15:01:57.080782 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7025,"thread_start_us":70,"threads_started":1}
I20260926 15:01:57.080907 20339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:57.068304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40646 (local address 127.8.102.3:36429)
0926 15:01:57.068916 (+ 612us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:57.068920 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:57.075122 (+ 6202us) server_negotiation.cc:407] Connection header received
0926 15:01:57.075175 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:57.075177 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:57.075193 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:57.075262 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:01:57.075990 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.077519 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.078894 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.080440 (+ 1546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.080514 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:57.080582 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:57.080775 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 15:01:57.080820 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":579,"thread_start_us":92,"threads_started":1}
I20260926 15:01:57.081439 20340 tablet_bootstrap.cc:492] T 911f75a3b574491db8827b558033527d P 0acb5ebcf0a242b1a83b07d117cf9611: Bootstrap starting.
I20260926 15:01:57.081926 20255 tablet_service.cc:1467] Processing CreateTablet for tablet 911f75a3b574491db8827b558033527d (DEFAULT_TABLE table=test-table [id=d564c86d08ac42088ac7e88c6c669a4e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:01:57.082274 20340 tablet_bootstrap.cc:654] T 911f75a3b574491db8827b558033527d P 0acb5ebcf0a242b1a83b07d117cf9611: Neither blocks nor log segments found. Creating new log.
I20260926 15:01:57.082265 20255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 911f75a3b574491db8827b558033527d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:01:57.082710 20340 log.cc:824] T 911f75a3b574491db8827b558033527d P 0acb5ebcf0a242b1a83b07d117cf9611: Log is configured to *not* fsync() on all Append() calls
I20260926 15:01:57.082975 20341 tablet_bootstrap.cc:492] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69: Bootstrap starting.
I20260926 15:01:57.084776 20341 tablet_bootstrap.cc:654] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69: Neither blocks nor log segments found. Creating new log.
I20260926 15:01:57.085191 20341 log.cc:824] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69: Log is configured to *not* fsync() on all Append() calls
I20260926 15:01:57.086108 20340 tablet_bootstrap.cc:492] T 911f75a3b574491db8827b558033527d P 0acb5ebcf0a242b1a83b07d117cf9611: No bootstrap required, opened a new log
I20260926 15:01:57.086191 20340 ts_tablet_manager.cc:1397] T 911f75a3b574491db8827b558033527d P 0acb5ebcf0a242b1a83b07d117cf9611: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260926 15:01:57.089470 20340 raft_consensus.cc:348] T 911f75a3b574491db8827b558033527d P 0acb5ebcf0a242b1a83b07d117cf9611 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c09ed35f8a914b308bedd0bbb786be69" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35093 } } peers { permanent_uuid: "0acb5ebcf0a242b1a83b07d117cf9611" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41923 } } peers { permanent_uuid: "e06c709a62c04118a7a11d55b1d1caba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36429 } }
I20260926 15:01:57.089686 20340 raft_consensus.cc:374] T 911f75a3b574491db8827b558033527d P 0acb5ebcf0a242b1a83b07d117cf9611 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:01:57.089726 20340 raft_consensus.cc:729] T 911f75a3b574491db8827b558033527d P 0acb5ebcf0a242b1a83b07d117cf9611 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0acb5ebcf0a242b1a83b07d117cf9611, State: Initialized, Role: FOLLOWER
I20260926 15:01:57.089874 20340 consensus_queue.cc:260] T 911f75a3b574491db8827b558033527d P 0acb5ebcf0a242b1a83b07d117cf9611 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c09ed35f8a914b308bedd0bbb786be69" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35093 } } peers { permanent_uuid: "0acb5ebcf0a242b1a83b07d117cf9611" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41923 } } peers { permanent_uuid: "e06c709a62c04118a7a11d55b1d1caba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36429 } }
I20260926 15:01:57.090189 20340 ts_tablet_manager.cc:1428] T 911f75a3b574491db8827b558033527d P 0acb5ebcf0a242b1a83b07d117cf9611: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 15:01:57.090330 20047 heartbeater.cc:503] Master 127.8.102.62:34227 was elected leader, sending a full tablet report...
I20260926 15:01:57.091097 20341 tablet_bootstrap.cc:492] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69: No bootstrap required, opened a new log
I20260926 15:01:57.091181 20341 ts_tablet_manager.cc:1397] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
I20260926 15:01:57.093276 20341 raft_consensus.cc:348] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c09ed35f8a914b308bedd0bbb786be69" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35093 } } peers { permanent_uuid: "0acb5ebcf0a242b1a83b07d117cf9611" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41923 } } peers { permanent_uuid: "e06c709a62c04118a7a11d55b1d1caba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36429 } }
I20260926 15:01:57.093489 20341 raft_consensus.cc:374] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:01:57.093538 20341 raft_consensus.cc:729] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c09ed35f8a914b308bedd0bbb786be69, State: Initialized, Role: FOLLOWER
I20260926 15:01:57.094352 20341 consensus_queue.cc:260] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c09ed35f8a914b308bedd0bbb786be69" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35093 } } peers { permanent_uuid: "0acb5ebcf0a242b1a83b07d117cf9611" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41923 } } peers { permanent_uuid: "e06c709a62c04118a7a11d55b1d1caba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36429 } }
I20260926 15:01:57.094473 20344 tablet_bootstrap.cc:492] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba: Bootstrap starting.
I20260926 15:01:57.094722 20341 ts_tablet_manager.cc:1428] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 15:01:57.094859 20186 heartbeater.cc:503] Master 127.8.102.62:34227 was elected leader, sending a full tablet report...
I20260926 15:01:57.095646 20344 tablet_bootstrap.cc:654] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba: Neither blocks nor log segments found. Creating new log.
I20260926 15:01:57.096028 20344 log.cc:824] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba: Log is configured to *not* fsync() on all Append() calls
I20260926 15:01:57.097055 20344 tablet_bootstrap.cc:492] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba: No bootstrap required, opened a new log
I20260926 15:01:57.097152 20344 ts_tablet_manager.cc:1397] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 15:01:57.099613 20344 raft_consensus.cc:348] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c09ed35f8a914b308bedd0bbb786be69" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35093 } } peers { permanent_uuid: "0acb5ebcf0a242b1a83b07d117cf9611" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41923 } } peers { permanent_uuid: "e06c709a62c04118a7a11d55b1d1caba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36429 } }
I20260926 15:01:57.099825 20344 raft_consensus.cc:374] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:01:57.099870 20344 raft_consensus.cc:729] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e06c709a62c04118a7a11d55b1d1caba, State: Initialized, Role: FOLLOWER
I20260926 15:01:57.100034 20344 consensus_queue.cc:260] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c09ed35f8a914b308bedd0bbb786be69" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35093 } } peers { permanent_uuid: "0acb5ebcf0a242b1a83b07d117cf9611" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41923 } } peers { permanent_uuid: "e06c709a62c04118a7a11d55b1d1caba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36429 } }
I20260926 15:01:57.100365 20344 ts_tablet_manager.cc:1428] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260926 15:01:57.100498 20321 heartbeater.cc:503] Master 127.8.102.62:34227 was elected leader, sending a full tablet report...
Sep 26 15:01:57 dist-test-slave-jnhs krb5kdc[19361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434916, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for HTTP/127.8.102.20@KRBTEST.COM
I20260926 15:01:57.208288 20346 raft_consensus.cc:482] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:01:57.208456 20346 raft_consensus.cc:504] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c09ed35f8a914b308bedd0bbb786be69" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35093 } } peers { permanent_uuid: "0acb5ebcf0a242b1a83b07d117cf9611" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41923 } } peers { permanent_uuid: "e06c709a62c04118a7a11d55b1d1caba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36429 } }
I20260926 15:01:57.208906 20346 leader_election.cc:288] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0acb5ebcf0a242b1a83b07d117cf9611 (127.8.102.1:41923), e06c709a62c04118a7a11d55b1d1caba (127.8.102.3:36429)
I20260926 15:01:57.213109 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:57.209168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:36429 (local address 127.8.102.2:56473)
0926 15:01:57.209229 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:57.209246 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:57.209314 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:57.209765 (+ 451us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:57.209766 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:57.209768 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:01:57.209850 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:57.209852 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.210865 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.210866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:57.212053 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:57.212056 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.212846 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.212848 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:57.212926 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:57.212931 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:01:57.212979 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 15:01:57.213025 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 15:01:57.213532 20339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:57.209290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:56473 (local address 127.8.102.3:36429)
0926 15:01:57.209334 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:57.209337 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:57.209347 (+ 10us) server_negotiation.cc:407] Connection header received
0926 15:01:57.209381 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:57.209382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:57.209391 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:57.209446 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:01:57.209941 (+ 495us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.210738 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.212159 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.212720 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.212788 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:57.212813 (+ 25us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:57.213404 (+ 591us) server_negotiation.cc:299] Negotiation successful
0926 15:01:57.213450 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260926 15:01:57.213896 20265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911f75a3b574491db8827b558033527d" candidate_uuid: "c09ed35f8a914b308bedd0bbb786be69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e06c709a62c04118a7a11d55b1d1caba" is_pre_election: true
I20260926 15:01:57.214061 20265 raft_consensus.cc:2454] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c09ed35f8a914b308bedd0bbb786be69 in term 0.
I20260926 15:01:57.214404 20073 leader_election.cc:302] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c09ed35f8a914b308bedd0bbb786be69, e06c709a62c04118a7a11d55b1d1caba; no voters:
I20260926 15:01:57.214546 20346 raft_consensus.cc:2781] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:01:57.214587 20346 raft_consensus.cc:482] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:01:57.214608 20346 raft_consensus.cc:3037] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:01:57.215585 20346 raft_consensus.cc:504] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c09ed35f8a914b308bedd0bbb786be69" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35093 } } peers { permanent_uuid: "0acb5ebcf0a242b1a83b07d117cf9611" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41923 } } peers { permanent_uuid: "e06c709a62c04118a7a11d55b1d1caba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36429 } }
I20260926 15:01:57.215726 20346 leader_election.cc:288] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [CANDIDATE]: Term 1 election: Requested vote from peers 0acb5ebcf0a242b1a83b07d117cf9611 (127.8.102.1:41923), e06c709a62c04118a7a11d55b1d1caba (127.8.102.3:36429)
I20260926 15:01:57.216499 20265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911f75a3b574491db8827b558033527d" candidate_uuid: "c09ed35f8a914b308bedd0bbb786be69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e06c709a62c04118a7a11d55b1d1caba"
I20260926 15:01:57.216596 20265 raft_consensus.cc:3037] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:01:57.217482 20265 raft_consensus.cc:2454] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c09ed35f8a914b308bedd0bbb786be69 in term 1.
I20260926 15:01:57.217672 20073 leader_election.cc:302] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c09ed35f8a914b308bedd0bbb786be69, e06c709a62c04118a7a11d55b1d1caba; no voters:
I20260926 15:01:57.217772 20346 raft_consensus.cc:2781] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:01:57.217936 20346 raft_consensus.cc:686] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [term 1 LEADER]: Becoming Leader. State: Replica: c09ed35f8a914b308bedd0bbb786be69, State: Running, Role: LEADER
I20260926 15:01:57.218039 20346 consensus_queue.cc:237] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c09ed35f8a914b308bedd0bbb786be69" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35093 } } peers { permanent_uuid: "0acb5ebcf0a242b1a83b07d117cf9611" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41923 } } peers { permanent_uuid: "e06c709a62c04118a7a11d55b1d1caba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36429 } }
I20260926 15:01:57.218953 19837 catalog_manager.cc:5495] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 reported cstate change: term changed from 0 to 1, leader changed from <none> to c09ed35f8a914b308bedd0bbb786be69 (127.8.102.2). New cstate: current_term: 1 leader_uuid: "c09ed35f8a914b308bedd0bbb786be69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c09ed35f8a914b308bedd0bbb786be69" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 35093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0acb5ebcf0a242b1a83b07d117cf9611" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41923 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e06c709a62c04118a7a11d55b1d1caba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36429 } health_report { overall_health: UNKNOWN } } }
I20260926 15:01:57.221063 20188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:57.209602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:41923 (local address 127.8.102.2:51265)
0926 15:01:57.213208 (+ 3606us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:57.213221 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:01:57.213269 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:01:57.213583 (+ 314us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:01:57.213584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:01:57.213586 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:01:57.213622 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:57.213624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.215259 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.215260 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:57.216321 (+ 1061us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:01:57.216323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.220828 (+ 4505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.220829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:01:57.220895 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:57.220899 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:01:57.220946 (+ 47us) client_negotiation.cc:240] Negotiation successful
0926 15:01:57.220992 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3594,"thread_start_us":116,"threads_started":1}
I20260926 15:01:57.221390 20336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:57.209673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:51265 (local address 127.8.102.1:41923)
0926 15:01:57.209716 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:57.209719 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:57.213439 (+ 3720us) server_negotiation.cc:407] Connection header received
0926 15:01:57.213470 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:57.213472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:57.213486 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:57.213537 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:01:57.214251 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.214939 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.220091 (+ 5152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.220705 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.220780 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:57.220791 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:57.221272 (+ 481us) server_negotiation.cc:299] Negotiation successful
0926 15:01:57.221330 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 15:01:57.221974 20001 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911f75a3b574491db8827b558033527d" candidate_uuid: "c09ed35f8a914b308bedd0bbb786be69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0acb5ebcf0a242b1a83b07d117cf9611" is_pre_election: true
I20260926 15:01:57.222131 20001 raft_consensus.cc:2454] T 911f75a3b574491db8827b558033527d P 0acb5ebcf0a242b1a83b07d117cf9611 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c09ed35f8a914b308bedd0bbb786be69 in term 0.
I20260926 15:01:57.222230 20000 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "911f75a3b574491db8827b558033527d" candidate_uuid: "c09ed35f8a914b308bedd0bbb786be69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0acb5ebcf0a242b1a83b07d117cf9611"
I20260926 15:01:57.222283 20000 raft_consensus.cc:3037] T 911f75a3b574491db8827b558033527d P 0acb5ebcf0a242b1a83b07d117cf9611 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:01:57.223193 20000 raft_consensus.cc:2454] T 911f75a3b574491db8827b558033527d P 0acb5ebcf0a242b1a83b07d117cf9611 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c09ed35f8a914b308bedd0bbb786be69 in term 1.
I20260926 15:01:57.254012 20338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:57.250542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34496 (local address 127.8.102.2:35093)
0926 15:01:57.250602 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 15:01:57.250605 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:01:57.250614 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:01:57.250679 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:01:57.250680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:01:57.250703 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:01:57.250780 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:01:57.250993 (+ 213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.251793 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.252954 (+ 1161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:01:57.253268 (+ 314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:01:57.253335 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:01:57.253529 (+ 194us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:01:57.253665 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:01:57.253735 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 15:01:57.253897 (+ 162us) server_negotiation.cc:299] Negotiation successful
0926 15:01:57.253914 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 15:01:57.256215 20000 raft_consensus.cc:1261] T 911f75a3b574491db8827b558033527d P 0acb5ebcf0a242b1a83b07d117cf9611 [term 1 FOLLOWER]: Refusing update from remote peer c09ed35f8a914b308bedd0bbb786be69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:01:57.256214 20265 raft_consensus.cc:1261] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba [term 1 FOLLOWER]: Refusing update from remote peer c09ed35f8a914b308bedd0bbb786be69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:01:57.256553 20346 consensus_queue.cc:1035] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e06c709a62c04118a7a11d55b1d1caba" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:01:57.256675 20350 consensus_queue.cc:1035] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0acb5ebcf0a242b1a83b07d117cf9611" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41923 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:01:57.264304 20353 mvcc.cc:205] Tried to move back new op lower bound from 7333621421078351872 to 7333621420925558784. Current Snapshot: MvccSnapshot[applied={T|T < 7333621421078351872}]
I20260926 15:01:57.265268 20355 mvcc.cc:205] Tried to move back new op lower bound from 7333621421078351872 to 7333621420925558784. Current Snapshot: MvccSnapshot[applied={T|T < 7333621421078351872}]
I20260926 15:01:57.265985 20354 mvcc.cc:205] Tried to move back new op lower bound from 7333621421078351872 to 7333621420925558784. Current Snapshot: MvccSnapshot[applied={T|T < 7333621421078351872}]
I20260926 15:01:57.273154 19836 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:01:57.273267 19836 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50114:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:01:57.274154 19836 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 313f03bf22304a31bd7d481b9b7729a7: Sending DeleteTablet for 3 replicas of tablet 911f75a3b574491db8827b558033527d
I20260926 15:01:57.274605 20118 tablet_service.cc:1514] Processing DeleteTablet for tablet 911f75a3b574491db8827b558033527d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:01:57 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:34480
I20260926 15:01:57.274873 20255 tablet_service.cc:1514] Processing DeleteTablet for tablet 911f75a3b574491db8827b558033527d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:01:57 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:40646
I20260926 15:01:57.275410 19973 tablet_service.cc:1514] Processing DeleteTablet for tablet 911f75a3b574491db8827b558033527d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:01:57 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:41510
I20260926 15:01:57.276124 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 19918
I20260926 15:01:57.279976 20363 tablet_replica.cc:330] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba: stopping tablet replica
I20260926 15:01:57.280157 20363 raft_consensus.cc:2229] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:01:57.280288 20363 raft_consensus.cc:2258] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:01:57.280486 20362 tablet_replica.cc:330] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69: stopping tablet replica
I20260926 15:01:57.280589 20362 raft_consensus.cc:2229] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:01:57.280716 20362 raft_consensus.cc:2258] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:01:57.281167 20363 ts_tablet_manager.cc:1905] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:01:57.282773 20363 ts_tablet_manager.cc:1918] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:01:57.282927 20363 log.cc:1196] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/911f75a3b574491db8827b558033527d
I20260926 15:01:57.283613 20363 ts_tablet_manager.cc:1939] T 911f75a3b574491db8827b558033527d P e06c709a62c04118a7a11d55b1d1caba: Deleting consensus metadata
I20260926 15:01:57.284611 19821 catalog_manager.cc:4841] TS e06c709a62c04118a7a11d55b1d1caba (127.8.102.3:36429): tablet 911f75a3b574491db8827b558033527d (table test-table [id=d564c86d08ac42088ac7e88c6c669a4e]) successfully deleted
I20260926 15:01:57.284857 20362 ts_tablet_manager.cc:1905] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:01:57.286392 20362 ts_tablet_manager.cc:1918] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:01:57.286533 20362 log.cc:1196] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/911f75a3b574491db8827b558033527d
I20260926 15:01:57.286896 20362 ts_tablet_manager.cc:1939] T 911f75a3b574491db8827b558033527d P c09ed35f8a914b308bedd0bbb786be69: Deleting consensus metadata
I20260926 15:01:57.287470 19821 catalog_manager.cc:4841] TS c09ed35f8a914b308bedd0bbb786be69 (127.8.102.2:35093): tablet 911f75a3b574491db8827b558033527d (table test-table [id=d564c86d08ac42088ac7e88c6c669a4e]) successfully deleted
W20260926 15:01:57.290110 19824 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.102.1:41923 (error 108)
I20260926 15:01:57.290366 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 20055
I20260926 15:01:57.293073 20337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:01:57.292927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:41923 (local address 127.0.0.1:41524)
0926 15:01:57.292984 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 15:01:57.293025 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.1:41923: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 15:01:57.293219 19824 catalog_manager.cc:4584] TS 0acb5ebcf0a242b1a83b07d117cf9611 (127.8.102.1:41923): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 911f75a3b574491db8827b558033527d: Network error: Client connection negotiation failed: client connection to 127.8.102.1:41923: connect: Connection refused (error 111)
I20260926 15:01:57.297073 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 20192
I20260926 15:01:57.303201 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 19805
I20260926 15:01:57.311228 8600 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 15:01:57.646246 8600 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 15:01:57.646402 8600 mini_ranger.cc:66] Stopping Ranger...
I20260926 15:01:57.991801 8600 mini_ranger.cc:68] Stopped Ranger
2026-09-26 15:01:57.992 UTC [19391] LOG: received smart shutdown request
2026-09-26 15:01:57.996 UTC [19391] LOG: background worker "logical replication launcher" (PID 19400) exited with exit code 1
2026-09-26 15:01:57.996 UTC [19395] LOG: shutting down
2026-09-26 15:01:58.215 UTC [19391] LOG: database system is shut down
2026-09-26T15:01:58Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (49861 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:01:58 dist-test-slave-jnhs krb5kdc[20374](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:01:58 dist-test-slave-jnhs krb5kdc[20374](info): set up 2 sockets
Sep 26 15:01:58 dist-test-slave-jnhs krb5kdc[20374](info): commencing operation
krb5kdc: starting...
W20260926 15:01:59.394830 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:01:59 dist-test-slave-jnhs krb5kdc[20374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434919, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:01:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:01:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.8.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.8.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.8.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/rangeradmin_127.8.102.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.8.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/rangeradmin_127.8.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.8.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.8.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.8.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/rangerlookup_127.8.102.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.8.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/rangerlookup_127.8.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.8.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/HTTP_127.8.102.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/HTTP_127.8.102.20@KRBTEST.COM.keytab.
I20260926 15:01:59.582252 8600 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-task17_aQt/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/postgres -l logfile start
2026-09-26 15:02:02.583 UTC [20404] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-26 15:02:02.583 UTC [20404] LOG: listening on IPv4 address "127.8.102.20", port 53703
2026-09-26 15:02:02.588 UTC [20404] LOG: listening on Unix socket "/tmp/.s.PGSQL.53703"
2026-09-26 15:02:02.624 UTC [20407] LOG: database system was shut down at 2026-09-26 15:02:00 UTC
2026-09-26 15:02:02.629 UTC [20404] LOG: database system is ready to accept connections
I20260926 15:02:03.582804 8600 mini_postgres.cc:96] Postgres bound to 53703
2026-09-26 15:02:03.589 UTC [20415] FATAL: database "slave" does not exist
127.8.102.20:53703 - accepting connections
I20260926 15:02:03.589862 8600 mini_ranger.cc:161] Starting Ranger...
I20260926 15:02:03.606101 8600 mini_ranger.cc:84] Created miniranger Postgres user
I20260926 15:02:04.262569 8600 mini_ranger.cc:87] Created ranger Postgres database
I20260926 15:02:04.262698 8600 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ranger-admin
2026-09-26 15:02:04,554 [I] DB FLAVOR :POSTGRES
2026-09-26 15:02:04,554 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 15:02:04,554 [I] Checking connection..
2026-09-26 15:02:04,555 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-26 15:02:04,813 [I] Checking connection passed.
2026-09-26 15:02:04,813 [I] --------- Verifying version history table ---------
2026-09-26 15:02:04,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 15:02:05,058 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 15:02:05,059 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 15:02:05,321 [I] Table x_db_version_h does not exist in database ranger
2026-09-26 15:02:05,321 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-26 15:02:05,321 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-26 15:02:05.590 UTC [20479] WARNING: there is no transaction in progress
2026-09-26 15:02:05,597 [I] create_dbversion_catalog.sql file imported successfully
2026-09-26 15:02:05,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-26 15:02:05,859 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-26 15:02:05,860 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 15:02:05,860 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-26 15:02:06,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-26 15:02:06,397 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-jnhs.c.gcp-upstream.internal','N') ;"
2026-09-26 15:02:06,650 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 15:02:06,902 [I] Table x_portal_user does not exist in database ranger
2026-09-26 15:02:06,903 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 15:02:07,160 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-26 15:02:07,161 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 15:02:07,400 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 15:02:07,643 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-26 15:02:07,644 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-26 15:02:08.800 UTC [20587] WARNING: there is no transaction in progress
2026-09-26 15:02:08.819 UTC [20587] WARNING: there is no transaction in progress
2026-09-26 15:02:08.840 UTC [20587] WARNING: there is no transaction in progress
2026-09-26 15:02:08.858 UTC [20587] WARNING: there is no transaction in progress
2026-09-26 15:02:08.877 UTC [20587] WARNING: there is no transaction in progress
2026-09-26 15:02:08.895 UTC [20587] WARNING: there is no transaction in progress
2026-09-26 15:02:08.956 UTC [20587] WARNING: there is no transaction in progress
2026-09-26 15:02:08.964 UTC [20587] WARNING: there is no transaction in progress
2026-09-26 15:02:08.980 UTC [20587] WARNING: there is no transaction in progress
2026-09-26 15:02:08.992 UTC [20587] WARNING: there is no transaction in progress
2026-09-26 15:02:09.004 UTC [20587] WARNING: there is no transaction in progress
2026-09-26 15:02:09.019 UTC [20587] WARNING: there is no transaction in progress
2026-09-26 15:02:09.030 UTC [20587] WARNING: there is no transaction in progress
2026-09-26 15:02:09.038 UTC [20587] WARNING: there is no transaction in progress
2026-09-26 15:02:09.047 UTC [20587] WARNING: there is no transaction in progress
2026-09-26 15:02:09.057 UTC [20587] WARNING: there is no transaction in progress
2026-09-26 15:02:09,935 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-26 15:02:09,936 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-26 15:02:10,175 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-26 15:02:10,175 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-26 15:02:10,426 [I] Table x_portal_user already exists in database 'ranger'
2026-09-26 15:02:10,427 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-26 15:02:10,701 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-26 15:02:10,702 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-26 15:02:10,974 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-26 15:02:11,222 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-jnhs.c.gcp-upstream.internal';"
2026-09-26 15:02:11,469 [I] CORE_DB_SCHEMA import status has been updated
2026-09-26 15:02:11,469 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-26 15:02:11,720 [I] DB_PATCHES have already been applied
I20260926 15:02:11.726744 8600 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ranger-admin:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-task17_aQt/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260926 15:02:11.726815 8600 mini_ranger.cc:193] Using host: 127.8.102.20
I20260926 15:02:11.729918 8600 mini_ranger.cc:227] Ranger admin URL: http://127.8.102.20:55337
Sep 26, 2026 3:02:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:02:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:02:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ranger-admin/ews/webapp
Sep 26, 2026 3:02:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 26, 2026 3:02:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 26, 2026 3:02:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ranger-admin/ews/webapp].
Sep 26, 2026 3:02:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260926 15:02:12.862031 8600 mini_ranger.cc:160] Time spent starting Ranger: real 9.272s user 0.001s sys 0.009s
Sep 26, 2026 3:02:34 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 26, 2026 3:02:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Sep 26, 2026 3:02:34 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 26, 2026 3:02:34 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 26, 2026 3:02:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 26, 2026 3:02:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 26, 2026 3:02:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 26, 2026 3:02:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 26, 2026 3:02:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 26, 2026 3:02:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 26, 2026 3:02:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 26, 2026 3:02:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 26, 2026 3:02:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 26, 2026 3:02:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 26, 2026 3:02:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 26, 2026 3:02:34 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 26, 2026 3:02:34 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 26, 2026 3:02:35 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260926 15:02:36.146283 8600 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.8.102.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.8.102.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.8.102.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/rangerkms_127.8.102.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.8.102.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/rangerkms_127.8.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.8.102.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.8.102.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.8.102.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/HTTP_127.8.102.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.8.102.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/HTTP_127.8.102.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 26 15:02:36 dist-test-slave-jnhs krb5kdc[20374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434956, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260926 15:02:36.240008 8600 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260926 15:02:36.250180 8600 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260926 15:02:37.281761 8600 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260926 15:02:37.281857 8600 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ranger-kms
I20260926 15:02:37.281879 8600 mini_ranger_kms.cc:227] Using postgres at 127.8.102.20:53703
2026-09-26 15:02:37,453 [I] DB FLAVOR :POSTGRES
2026-09-26 15:02:37,453 [I] --------- Verifying Ranger DB connection ---------
2026-09-26 15:02:37,453 [I] Checking connection
2026-09-26 15:02:37,454 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-26 15:02:37,735 [I] connection success
2026-09-26 15:02:37,735 [I] --------- Verifying Ranger DB tables ---------
2026-09-26 15:02:37,735 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-26 15:02:37,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-26 15:02:37,981 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-26 15:02:37,981 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-26 15:02:37,982 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-26 15:02:37,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.8.102.20:53703/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-26 15:02:38,259 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260926 15:02:38.836568 8600 mini_ranger_kms.cc:310] Created kms service
I20260926 15:02:39.055145 8600 mini_ranger_kms.cc:326] Created kudu user
I20260926 15:02:39.100893 8600 mini_ranger_kms.cc:343] Created rangerkms user
Sep 26 15:02:39 dist-test-slave-jnhs krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434956, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.8.102.20@KRBTEST.COM
I20260926 15:02:39.542966 8600 mini_ranger_kms.cc:384] Added ranger policy
I20260926 15:02:39.543211 8600 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ranger-kms:/tmp/dist-test-task17_aQt/build/release/bin/postgresql.jar:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-task17_aQt/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260926 15:02:39.543241 8600 mini_ranger_kms.cc:240] Using host: 127.8.102.20
I20260926 15:02:39.546872 8600 mini_ranger_kms.cc:276] Ranger KMS PID: 20768
I20260926 15:02:39.546967 8600 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.8.102.20:59047
Sep 26, 2026 3:02:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 26, 2026 3:02:40 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 26, 2026 3:02:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 26, 2026 3:02:40 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 26, 2026 3:02:41 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ranger-kms/ews/webapp].
Sep 26, 2026 3:02:41 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Sep 26 15:02:41 dist-test-slave-jnhs krb5kdc[20374](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790434961, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.102.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:41.883746 8600 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.644s user 0.007s sys 0.145s
I20260926 15:02:41.883867 8600 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260926 15:02:41.883917 8600 mini_ranger_kms.cc:401] 127.8.102.20:59047/kms/v1/keys
Sep 26 15:02:42 dist-test-slave-jnhs krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434961, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.102.20@KRBTEST.COM for HTTP/127.8.102.20@KRBTEST.COM
Sep 26 15:02:42 dist-test-slave-jnhs krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434961, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.8.102.20@KRBTEST.COM for HTTP/127.8.102.20@KRBTEST.COM
Sep 26 15:02:44 dist-test-slave-jnhs krb5kdc[20374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:02:44.977820 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:36139
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:36139
--ranger_config_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.102.20:59047/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:02:45.063787 20818 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:45.064049 20818 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:45.064090 20818 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:45.065372 20818 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:02:45.065448 20818 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:45.065474 20818 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:45.065502 20818 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:02:45.065534 20818 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:02:45.067242 20818 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/wal
--ranger_kms_url=127.8.102.20:59047/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:36139
--ranger_config_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:36139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20818
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:45.067528 20818 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:45.067772 20818 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:45.074331 20824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:45.074354 20818 server_base.cc:1034] running on GCE node
W20260926 15:02:45.074337 20823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:45.074596 20826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:45.074915 20818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:45.075258 20818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:45.076417 20818 hybrid_clock.cc:648] HybridClock initialized: now 1790434965076393 us; error 43 us; skew 500 ppm
Sep 26 15:02:45 dist-test-slave-jnhs krb5kdc[20374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434965, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:45.079252 20818 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:02:45.079881 20818 webserver.cc:460] Webserver started at http://127.8.102.62:34097/ using document root <none> and password file <none>
I20260926 15:02:45.080219 20818 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:45.080289 20818 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:02:45.080428 20818 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:02:45 dist-test-slave-jnhs krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434965, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for HTTP/127.8.102.20@KRBTEST.COM
I20260926 15:02:45.146601 20818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "7d5ddd535b8049c2a7c59d785aa03f02"
format_stamp: "Formatted at 2026-09-26 15:02:45 on dist-test-slave-jnhs"
server_key: "2839c33e56ebf3323992cb50669b3a0d"
server_key_iv: "b9b15ab733d976470a317b7c7efe0064"
server_key_version: "kuduclusterkey@0"
I20260926 15:02:45.147182 20818 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "7d5ddd535b8049c2a7c59d785aa03f02"
format_stamp: "Formatted at 2026-09-26 15:02:45 on dist-test-slave-jnhs"
server_key: "2839c33e56ebf3323992cb50669b3a0d"
server_key_iv: "b9b15ab733d976470a317b7c7efe0064"
server_key_version: "kuduclusterkey@0"
I20260926 15:02:45.148854 20818 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 15:02:45.164943 20833 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:45.165234 20818 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:45.165391 20818 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "7d5ddd535b8049c2a7c59d785aa03f02"
format_stamp: "Formatted at 2026-09-26 15:02:45 on dist-test-slave-jnhs"
server_key: "2839c33e56ebf3323992cb50669b3a0d"
server_key_iv: "b9b15ab733d976470a317b7c7efe0064"
server_key_version: "kuduclusterkey@0"
I20260926 15:02:45.165539 20818 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:45.176657 20818 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:45.178977 20818 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:45.179157 20818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:45.184124 20818 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:36139
I20260926 15:02:45.184149 20885 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:36139 every 8 connection(s)
I20260926 15:02:45.184602 20818 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:02:45.184767 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 20818
I20260926 15:02:45.184886 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:02:45.185111 8600 external_mini_cluster.cc:1428] Setting key 0213e9147cc1d91813b8e17a4cb11027
I20260926 15:02:45.185545 20886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:02:45.189241 20886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02: Bootstrap starting.
I20260926 15:02:45.190062 20886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02: Neither blocks nor log segments found. Creating new log.
I20260926 15:02:45.190369 20886 log.cc:824] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02: Log is configured to *not* fsync() on all Append() calls
I20260926 15:02:45.191241 20886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02: No bootstrap required, opened a new log
Sep 26 15:02:45 dist-test-slave-jnhs krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434964, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:02:45.193373 20886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5ddd535b8049c2a7c59d785aa03f02" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36139 } }
I20260926 15:02:45.193559 20886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:02:45.193611 20886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d5ddd535b8049c2a7c59d785aa03f02, State: Initialized, Role: FOLLOWER
I20260926 15:02:45.193763 20886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5ddd535b8049c2a7c59d785aa03f02" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36139 } }
I20260926 15:02:45.193838 20886 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:02:45.193861 20886 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:02:45.193888 20886 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:02:45.194492 20886 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5ddd535b8049c2a7c59d785aa03f02" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36139 } }
I20260926 15:02:45.194591 20886 leader_election.cc:302] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d5ddd535b8049c2a7c59d785aa03f02; no voters:
I20260926 15:02:45.194809 20886 leader_election.cc:288] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:02:45.195087 20886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:02:45.196612 20889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:45.185654 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40284 (local address 127.8.102.62:36139)
0926 15:02:45.185937 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:45.185945 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:45.185969 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:02:45.186397 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:45.186403 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:45.186449 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:45.186601 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:45.187239 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:45.188130 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:45.188919 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:45.189335 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:45.189466 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:45.192341 (+ 2875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:45.192346 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:45.192346 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:45.192360 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:45.194412 (+ 2052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:45.194889 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:45.194892 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:45.194893 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:45.194990 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:45.195257 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:45.195259 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:45.195259 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:45.195576 (+ 317us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:45.195747 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:45.196270 (+ 523us) server_negotiation.cc:299] Negotiation successful
0926 15:02:45.196392 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":240,"thread_start_us":87,"threads_started":1}
I20260926 15:02:45.196635 20886 ranger_client.cc:318] Using new properties file: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 15:02:45.196961 20892 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:02:45.197132 20892 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [term 1 LEADER]: Becoming Leader. State: Replica: 7d5ddd535b8049c2a7c59d785aa03f02, State: Running, Role: LEADER
I20260926 15:02:45.199229 20892 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5ddd535b8049c2a7c59d785aa03f02" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36139 } }
I20260926 15:02:45.200191 20892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7d5ddd535b8049c2a7c59d785aa03f02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5ddd535b8049c2a7c59d785aa03f02" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36139 } } }
I20260926 15:02:45.200487 20892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [sys.catalog]: This master's current role is: LEADER
I20260926 15:02:45.200812 20891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d5ddd535b8049c2a7c59d785aa03f02. Latest consensus state: current_term: 1 leader_uuid: "7d5ddd535b8049c2a7c59d785aa03f02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5ddd535b8049c2a7c59d785aa03f02" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36139 } } }
I20260926 15:02:45.200894 20891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [sys.catalog]: This master's current role is: LEADER
I20260926 15:02:45.200937 20907 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:02:45.201577 20907 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:02:45.203745 20907 catalog_manager.cc:1347] Generated new cluster ID: 4d21bb0713a341caab3a7acdc471c3e8
I20260926 15:02:45.203799 20907 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:02:45.222724 20907 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:02:45.224084 20907 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:02:45.250892 20907 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02: Generated new TSK 0
I20260926 15:02:45.251228 20907 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
I20260926 15:02:46.011825 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.102.20:59047/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36139
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:02:46.123618 20931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:46.123904 20931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:46.123945 20931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:46.125139 20931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:46.125218 20931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:46.125267 20931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:02:46.126909 20931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.8.102.20:59047/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20931
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:46.127188 20931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:46.127444 20931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:46.133019 20939 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:46.133025 20936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:46.133270 20937 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:46.133596 20931 server_base.cc:1034] running on GCE node
I20260926 15:02:46.133807 20931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:46.134085 20931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:46.135267 20931 hybrid_clock.cc:648] HybridClock initialized: now 1790434966135255 us; error 26 us; skew 500 ppm
Sep 26 15:02:46 dist-test-slave-jnhs krb5kdc[20374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434966, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:46.137876 20931 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:02:46.138387 20931 webserver.cc:460] Webserver started at http://127.8.102.1:42347/ using document root <none> and password file <none>
I20260926 15:02:46.138669 20931 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:46.138738 20931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:02:46.138891 20931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:02:46 dist-test-slave-jnhs krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434966, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for HTTP/127.8.102.20@KRBTEST.COM
I20260926 15:02:46.152380 20931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "3b48aa638ca74d34b25ec2d0c008bfc1"
format_stamp: "Formatted at 2026-09-26 15:02:46 on dist-test-slave-jnhs"
server_key: "2fbed1dfc2f88518c44b6e5a68c026d3"
server_key_iv: "a8c02ef7a68ab019b22bf5c68571656a"
server_key_version: "kuduclusterkey@0"
I20260926 15:02:46.152964 20931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "3b48aa638ca74d34b25ec2d0c008bfc1"
format_stamp: "Formatted at 2026-09-26 15:02:46 on dist-test-slave-jnhs"
server_key: "2fbed1dfc2f88518c44b6e5a68c026d3"
server_key_iv: "a8c02ef7a68ab019b22bf5c68571656a"
server_key_version: "kuduclusterkey@0"
I20260926 15:02:46.155647 20931 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260926 15:02:46.177018 20948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:46.177271 20931 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:46.177361 20931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "3b48aa638ca74d34b25ec2d0c008bfc1"
format_stamp: "Formatted at 2026-09-26 15:02:46 on dist-test-slave-jnhs"
server_key: "2fbed1dfc2f88518c44b6e5a68c026d3"
server_key_iv: "a8c02ef7a68ab019b22bf5c68571656a"
server_key_version: "kuduclusterkey@0"
I20260926 15:02:46.177451 20931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:46.198143 20931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:46.200572 20931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:46.200819 20931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:46.201145 20931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:02:46.201586 20931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:02:46.201699 20931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:46.201804 20931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:02:46.201884 20931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:46.208477 20931 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:33757
I20260926 15:02:46.209048 20931 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:02:46.210296 21061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:33757 every 8 connection(s)
I20260926 15:02:46.219287 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 20931
I20260926 15:02:46.219419 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:02:46.219633 8600 external_mini_cluster.cc:1428] Setting key 0594fbf5e8d2af32ee61447042ea0cf9
Sep 26 15:02:46 dist-test-slave-jnhs krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434966, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:02:46.226989 21065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:46.211198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36139 (local address 127.8.102.1:42407)
0926 15:02:46.214806 (+ 3608us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:46.214866 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:46.215523 (+ 657us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:46.216064 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:46.216070 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:46.216345 (+ 275us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:46.216635 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:46.216642 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:46.217793 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:46.217794 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:46.219130 (+ 1336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:46.219134 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:46.219915 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:46.219916 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:46.220072 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:46.220729 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:46.220740 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:46.222414 (+ 1674us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:46.224742 (+ 2328us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:46.224752 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:46.224755 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:46.225087 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:46.225329 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:46.225329 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:46.225330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:46.225475 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:46.225849 (+ 374us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:46.225857 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:46.226075 (+ 218us) client_negotiation.cc:769] Sending connection context
0926 15:02:46.226228 (+ 153us) client_negotiation.cc:240] Negotiation successful
0926 15:02:46.226480 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3550,"thread_start_us":86,"threads_started":1}
I20260926 15:02:46.227653 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:46.210996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:42407 (local address 127.8.102.62:36139)
0926 15:02:46.215774 (+ 4778us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:46.215776 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:46.215785 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:02:46.215813 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:46.215814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:46.215827 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:46.215894 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:46.216763 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:46.217678 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:46.219244 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:46.219755 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:46.219827 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:46.222563 (+ 2736us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:46.222565 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:46.222573 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:46.222582 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:46.224621 (+ 2039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:46.225184 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:46.225186 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:46.225186 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:46.225256 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:46.225548 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:46.225549 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:46.225549 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:46.225767 (+ 218us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:46.227409 (+ 1642us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:46.227459 (+ 50us) server_negotiation.cc:299] Negotiation successful
0926 15:02:46.227557 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4741,"thread_start_us":81,"threads_started":1}
I20260926 15:02:46.228166 21062 heartbeater.cc:347] Connected to a master server at 127.8.102.62:36139
I20260926 15:02:46.228271 21062 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:46.228477 21062 heartbeater.cc:511] Master 127.8.102.62:36139 requested a full tablet report, sending...
I20260926 15:02:46.229077 20850 ts_manager.cc:194] Registered new tserver with Master: 3b48aa638ca74d34b25ec2d0c008bfc1 (127.8.102.1:33757)
I20260926 15:02:46.230036 20850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:42407
WARNING: no policy specified for kudu/127.8.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
I20260926 15:02:46.258134 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.102.20:59047/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36139
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:02:46 dist-test-slave-jnhs krb5kdc[20374](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790434966, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260926 15:02:46.393443 21068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:46.393811 21068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:46.393888 21068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:46.396174 21068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:46.396296 21068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:46.396375 21068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:02:46.398874 21068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.8.102.20:59047/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:46.399362 21068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:46.399688 21068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:46.406262 21076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:46.406725 21075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:46.406946 21078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:46.408046 21068 server_base.cc:1034] running on GCE node
I20260926 15:02:46.408347 21068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:46.408639 21068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:46.409830 21068 hybrid_clock.cc:648] HybridClock initialized: now 1790434966409811 us; error 34 us; skew 500 ppm
Sep 26 15:02:46 dist-test-slave-jnhs krb5kdc[20374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434966, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:46.413103 21068 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:02:46.413609 21068 webserver.cc:460] Webserver started at http://127.8.102.2:42003/ using document root <none> and password file <none>
I20260926 15:02:46.413869 21068 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:46.413919 21068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:02:46.414041 21068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:02:46 dist-test-slave-jnhs krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434966, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for HTTP/127.8.102.20@KRBTEST.COM
I20260926 15:02:46.433815 21068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "86a3206390894b388647e4673be4d2be"
format_stamp: "Formatted at 2026-09-26 15:02:46 on dist-test-slave-jnhs"
server_key: "a2d9652d21e8a59c6e40954203454091"
server_key_iv: "d636dbf70a4d09dd0e4963d5db0a5ff4"
server_key_version: "kuduclusterkey@0"
I20260926 15:02:46.434379 21068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "86a3206390894b388647e4673be4d2be"
format_stamp: "Formatted at 2026-09-26 15:02:46 on dist-test-slave-jnhs"
server_key: "a2d9652d21e8a59c6e40954203454091"
server_key_iv: "d636dbf70a4d09dd0e4963d5db0a5ff4"
server_key_version: "kuduclusterkey@0"
I20260926 15:02:46.443351 21068 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.000s sys 0.001s
I20260926 15:02:46.462605 21085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:46.462951 21068 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.001s
I20260926 15:02:46.463053 21068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "86a3206390894b388647e4673be4d2be"
format_stamp: "Formatted at 2026-09-26 15:02:46 on dist-test-slave-jnhs"
server_key: "a2d9652d21e8a59c6e40954203454091"
server_key_iv: "d636dbf70a4d09dd0e4963d5db0a5ff4"
server_key_version: "kuduclusterkey@0"
I20260926 15:02:46.463156 21068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:46.477869 21068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:46.480237 21068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:46.480383 21068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:46.480589 21068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:02:46.480957 21068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:02:46.480995 21068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:46.481027 21068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:02:46.481048 21068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:46.487535 21068 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:37313
I20260926 15:02:46.488055 21068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:02:46.490916 21199 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:37313 every 8 connection(s)
I20260926 15:02:46.495734 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 21068
I20260926 15:02:46.495841 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:02:46.496025 8600 external_mini_cluster.cc:1428] Setting key 88f34f070bc28fb6446abf68296f6abb
Sep 26 15:02:46 dist-test-slave-jnhs krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434966, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:02:46.502755 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:46.491635 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:56075 (local address 127.8.102.62:36139)
0926 15:02:46.491694 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:46.491696 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:46.492209 (+ 513us) server_negotiation.cc:407] Connection header received
0926 15:02:46.492925 (+ 716us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:46.492926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:46.492939 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:46.492969 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:46.493805 (+ 836us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:46.494611 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:46.496155 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:46.496448 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:46.496489 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:46.499319 (+ 2830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:46.499321 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:46.499321 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:46.499329 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:46.501066 (+ 1737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:46.501574 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:46.501575 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:46.501575 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:46.501635 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:46.501939 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:46.501940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:46.501940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:46.502084 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:46.502130 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:46.502533 (+ 403us) server_negotiation.cc:299] Negotiation successful
0926 15:02:46.502624 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 15:02:46.503520 21202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:46.491623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36139 (local address 127.8.102.2:56075)
0926 15:02:46.492081 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:46.492123 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:46.492798 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:46.493153 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:46.493157 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:46.493410 (+ 253us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:46.493693 (+ 283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:46.493699 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:46.494764 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:46.494764 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:46.496023 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:46.496026 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:46.496589 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:46.496589 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:46.496730 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:46.497354 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:46.497364 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:46.499210 (+ 1846us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:46.501170 (+ 1960us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:46.501177 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:46.501179 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:46.501480 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:46.501718 (+ 238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:46.501718 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:46.501719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:46.501855 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:46.502181 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:46.502187 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:46.502368 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 15:02:46.503099 (+ 731us) client_negotiation.cc:240] Negotiation successful
0926 15:02:46.503274 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":418,"thread_start_us":110,"threads_started":1}
I20260926 15:02:46.504245 21200 heartbeater.cc:347] Connected to a master server at 127.8.102.62:36139
I20260926 15:02:46.504340 21200 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:46.504522 21200 heartbeater.cc:511] Master 127.8.102.62:36139 requested a full tablet report, sending...
I20260926 15:02:46.504976 20850 ts_manager.cc:194] Registered new tserver with Master: 86a3206390894b388647e4673be4d2be (127.8.102.2:37313)
I20260926 15:02:46.505455 20850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:56075
WARNING: no policy specified for kudu/127.8.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
I20260926 15:02:46.539048 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.102.20:59047/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36139
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:02:46.659178 21205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:46.659425 21205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:46.659464 21205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:46.660686 21205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:46.660759 21205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:46.660808 21205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:02:46.662374 21205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.8.102.20:59047/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21205
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:46.662711 21205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:46.662972 21205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:46.668085 21211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:46.668251 21213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:46.668061 21210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:46.668908 21205 server_base.cc:1034] running on GCE node
I20260926 15:02:46.669153 21205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:46.669451 21205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:46.670626 21205 hybrid_clock.cc:648] HybridClock initialized: now 1790434966670602 us; error 38 us; skew 500 ppm
Sep 26 15:02:46 dist-test-slave-jnhs krb5kdc[20374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434966, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:46.673779 21205 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:02:46.674295 21205 webserver.cc:460] Webserver started at http://127.8.102.3:43993/ using document root <none> and password file <none>
I20260926 15:02:46.674553 21205 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:46.674625 21205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:02:46.674832 21205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 26 15:02:46 dist-test-slave-jnhs krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434966, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for HTTP/127.8.102.20@KRBTEST.COM
I20260926 15:02:46.689944 21205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "8dd7eb3419bc4dbd86fdcb1ac83ba6d0"
format_stamp: "Formatted at 2026-09-26 15:02:46 on dist-test-slave-jnhs"
server_key: "dee779b93c8a977552fe08004af36ebb"
server_key_iv: "1118027c2d61fc1d54c07d2c91fc67ac"
server_key_version: "kuduclusterkey@0"
I20260926 15:02:46.690415 21205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "8dd7eb3419bc4dbd86fdcb1ac83ba6d0"
format_stamp: "Formatted at 2026-09-26 15:02:46 on dist-test-slave-jnhs"
server_key: "dee779b93c8a977552fe08004af36ebb"
server_key_iv: "1118027c2d61fc1d54c07d2c91fc67ac"
server_key_version: "kuduclusterkey@0"
I20260926 15:02:46.692480 21205 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.000s
I20260926 15:02:46.718361 21220 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:46.718859 21205 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.000s sys 0.000s
I20260926 15:02:46.718973 21205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "8dd7eb3419bc4dbd86fdcb1ac83ba6d0"
format_stamp: "Formatted at 2026-09-26 15:02:46 on dist-test-slave-jnhs"
server_key: "dee779b93c8a977552fe08004af36ebb"
server_key_iv: "1118027c2d61fc1d54c07d2c91fc67ac"
server_key_version: "kuduclusterkey@0"
I20260926 15:02:46.719074 21205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:46.768672 21205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:46.771389 21205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:46.771631 21205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:46.771911 21205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:02:46.772370 21205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:02:46.772475 21205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:46.772554 21205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:02:46.772621 21205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:46.780654 21205 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:42633
I20260926 15:02:46.781203 21205 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:02:46.782341 21333 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:42633 every 8 connection(s)
I20260926 15:02:46.788614 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 21205
I20260926 15:02:46.788877 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:02:46.789119 8600 external_mini_cluster.cc:1428] Setting key f4cd539316a0bd5f78d4222a60d94491
Sep 26 15:02:46 dist-test-slave-jnhs krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434966, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:02:46.795073 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:46.782600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:56615 (local address 127.8.102.62:36139)
0926 15:02:46.782685 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:46.782687 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:46.783103 (+ 416us) server_negotiation.cc:407] Connection header received
0926 15:02:46.783993 (+ 890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:46.783995 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:46.784009 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:46.784042 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:46.784931 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:46.785700 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:46.787209 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:46.787505 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:46.787554 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:46.791478 (+ 3924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:46.791479 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:46.791480 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:46.791488 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:46.793306 (+ 1818us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:46.793876 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:46.793877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:46.793878 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:46.793944 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:46.794238 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:46.794239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:46.794240 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:46.794387 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:46.794444 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:46.794888 (+ 444us) server_negotiation.cc:299] Negotiation successful
0926 15:02:46.794976 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 15:02:46.795861 21336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:46.782601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36139 (local address 127.8.102.3:56615)
0926 15:02:46.782962 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:46.783012 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:46.783857 (+ 845us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:46.784243 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:46.784248 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:46.784505 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:46.784811 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:46.784817 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:46.785821 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:46.785822 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:46.787099 (+ 1277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:46.787102 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:46.787662 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:46.787663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:46.787830 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:46.789628 (+ 1798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:46.789640 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:46.791355 (+ 1715us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:46.793415 (+ 2060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:46.793424 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:46.793427 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:46.793775 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:46.794016 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:46.794017 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:46.794017 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:46.794156 (+ 139us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:46.794480 (+ 324us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:46.794487 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:46.794713 (+ 226us) client_negotiation.cc:769] Sending connection context
0926 15:02:46.795416 (+ 703us) client_negotiation.cc:240] Negotiation successful
0926 15:02:46.795615 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":124,"threads_started":1}
I20260926 15:02:46.796828 21334 heartbeater.cc:347] Connected to a master server at 127.8.102.62:36139
I20260926 15:02:46.796942 21334 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:46.797145 21334 heartbeater.cc:511] Master 127.8.102.62:36139 requested a full tablet report, sending...
I20260926 15:02:46.797616 20850 ts_manager.cc:194] Registered new tserver with Master: 8dd7eb3419bc4dbd86fdcb1ac83ba6d0 (127.8.102.3:42633)
I20260926 15:02:46.798120 20850 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:56615
I20260926 15:02:46.800872 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:02:46.801720 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 20931
I20260926 15:02:46.808815 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 21068
I20260926 15:02:46.814447 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 21205
I20260926 15:02:46.818711 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 20818
I20260926 15:02:46.823415 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:36139
--webserver_interface=127.8.102.62
--webserver_port=34097
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:36139
--ranger_config_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.102.20:59047/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.102.20:59047/kms with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:02:46.906323 21337 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:46.906531 21337 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:46.906553 21337 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:46.907759 21337 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:02:46.907805 21337 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:46.907819 21337 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:46.907832 21337 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:02:46.907845 21337 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:02:46.909415 21337 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.102.20:59047/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:36139
--ranger_config_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:36139
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=34097
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21337
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:46.909658 21337 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:46.909915 21337 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:46.914858 21342 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:46.914948 21343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:46.914870 21345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:46.915238 21337 server_base.cc:1034] running on GCE node
I20260926 15:02:46.915496 21337 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:46.915752 21337 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:46.916935 21337 hybrid_clock.cc:648] HybridClock initialized: now 1790434966916926 us; error 38 us; skew 500 ppm
Sep 26 15:02:46 dist-test-slave-jnhs krb5kdc[20374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434966, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:46.919570 21337 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:02:46.920035 21337 webserver.cc:460] Webserver started at http://127.8.102.62:34097/ using document root <none> and password file <none>
I20260926 15:02:46.920279 21337 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:46.920356 21337 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 15:02:46 dist-test-slave-jnhs krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434966, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for HTTP/127.8.102.20@KRBTEST.COM
I20260926 15:02:46.979225 21337 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:46.980077 21353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:46.980311 21337 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:46.980419 21337 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "7d5ddd535b8049c2a7c59d785aa03f02"
format_stamp: "Formatted at 2026-09-26 15:02:45 on dist-test-slave-jnhs"
server_key: "2839c33e56ebf3323992cb50669b3a0d"
server_key_iv: "b9b15ab733d976470a317b7c7efe0064"
server_key_version: "kuduclusterkey@0"
I20260926 15:02:46.980881 21337 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:47.012338 21337 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:47.014755 21337 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:47.014919 21337 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:47.019981 21337 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:36139
I20260926 15:02:47.020015 21405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:36139 every 8 connection(s)
I20260926 15:02:47.020494 21337 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:02:47.021747 21406 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:02:47.022430 21406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02: Bootstrap starting.
I20260926 15:02:47.029752 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 21337
I20260926 15:02:47.030144 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.102.20:59047/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:33757
--local_ip_for_outbound_sockets=127.8.102.1
--tserver_master_addrs=127.8.102.62:36139
--webserver_port=42347
--webserver_interface=127.8.102.1
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.102.20:59047/kms with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:02:47.030810 21406 log.cc:824] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02: Log is configured to *not* fsync() on all Append() calls
I20260926 15:02:47.033208 21406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:02:47.033540 21406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02: Bootstrap complete.
I20260926 15:02:47.036934 21406 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5ddd535b8049c2a7c59d785aa03f02" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36139 } }
I20260926 15:02:47.037286 21406 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d5ddd535b8049c2a7c59d785aa03f02, State: Initialized, Role: FOLLOWER
I20260926 15:02:47.037503 21406 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5ddd535b8049c2a7c59d785aa03f02" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36139 } }
I20260926 15:02:47.037585 21406 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:02:47.037628 21406 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:02:47.037711 21406 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [term 1 FOLLOWER]: Advancing to term 2
I20260926 15:02:47.039283 21406 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5ddd535b8049c2a7c59d785aa03f02" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36139 } }
I20260926 15:02:47.039407 21406 leader_election.cc:302] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7d5ddd535b8049c2a7c59d785aa03f02; no voters:
I20260926 15:02:47.039628 21406 leader_election.cc:288] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 15:02:47.039707 21410 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 15:02:47.039950 21410 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [term 2 LEADER]: Becoming Leader. State: Replica: 7d5ddd535b8049c2a7c59d785aa03f02, State: Running, Role: LEADER
I20260926 15:02:47.040134 21410 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5ddd535b8049c2a7c59d785aa03f02" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36139 } }
I20260926 15:02:47.039981 21406 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:02:47.040553 21411 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "7d5ddd535b8049c2a7c59d785aa03f02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5ddd535b8049c2a7c59d785aa03f02" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36139 } } }
I20260926 15:02:47.040791 21411 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [sys.catalog]: This master's current role is: LEADER
I20260926 15:02:47.040727 21412 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7d5ddd535b8049c2a7c59d785aa03f02. Latest consensus state: current_term: 2 leader_uuid: "7d5ddd535b8049c2a7c59d785aa03f02" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7d5ddd535b8049c2a7c59d785aa03f02" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36139 } } }
I20260926 15:02:47.041039 21412 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02 [sys.catalog]: This master's current role is: LEADER
I20260926 15:02:47.041707 21414 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:02:47.042162 21406 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260926 15:02:47.042325 21414 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:02:47.042523 21414 catalog_manager.cc:1259] Loaded cluster ID: 4d21bb0713a341caab3a7acdc471c3e8
I20260926 15:02:47.042598 21414 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:02:47.043892 21414 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:02:47.044135 21414 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02: Loaded TSK: 0
I20260926 15:02:47.044440 21414 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 15:02:47.149463 21409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:47.149866 21409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:47.149971 21409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:47.152103 21409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:47.152384 21409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:47.152505 21409 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:02:47.155048 21409 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.102.20:59047/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:33757
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=42347
--tserver_master_addrs=127.8.102.62:36139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21409
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:47.155609 21409 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:47.155961 21409 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:47.162621 21444 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:47.163085 21443 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:47.162796 21446 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:47.164458 21409 server_base.cc:1034] running on GCE node
I20260926 15:02:47.164743 21409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:47.165088 21409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:47.166349 21409 hybrid_clock.cc:648] HybridClock initialized: now 1790434967166313 us; error 49 us; skew 500 ppm
Sep 26 15:02:47 dist-test-slave-jnhs krb5kdc[20374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434967, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:47.169919 21409 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:02:47.170480 21409 webserver.cc:460] Webserver started at http://127.8.102.1:42347/ using document root <none> and password file <none>
I20260926 15:02:47.170769 21409 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:47.170846 21409 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 15:02:47 dist-test-slave-jnhs krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434967, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for HTTP/127.8.102.20@KRBTEST.COM
I20260926 15:02:47.190152 21409 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.000s sys 0.001s
I20260926 15:02:47.191270 21454 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:47.191488 21409 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:47.191584 21409 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "3b48aa638ca74d34b25ec2d0c008bfc1"
format_stamp: "Formatted at 2026-09-26 15:02:46 on dist-test-slave-jnhs"
server_key: "2fbed1dfc2f88518c44b6e5a68c026d3"
server_key_iv: "a8c02ef7a68ab019b22bf5c68571656a"
server_key_version: "kuduclusterkey@0"
I20260926 15:02:47.192054 21409 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:47.214776 21409 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:47.217401 21409 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:47.217660 21409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:47.217938 21409 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:02:47.218355 21409 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:02:47.218451 21409 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:47.218572 21409 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:02:47.218700 21409 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:47.225409 21409 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:33757
I20260926 15:02:47.226066 21409 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:02:47.227406 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 21409
I20260926 15:02:47.227726 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.102.20:59047/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:37313
--local_ip_for_outbound_sockets=127.8.102.2
--tserver_master_addrs=127.8.102.62:36139
--webserver_port=42003
--webserver_interface=127.8.102.2
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.102.20:59047/kms with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:02:47.229450 21567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:33757 every 8 connection(s)
Sep 26 15:02:47 dist-test-slave-jnhs krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434967, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:02:47.253957 21571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:47.230170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36139 (local address 127.8.102.1:44901)
0926 15:02:47.230548 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:47.230603 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:47.231269 (+ 666us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:47.233287 (+ 2018us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:47.233291 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:47.233527 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:47.233824 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:47.233832 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:47.235521 (+ 1689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:47.235521 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:47.236934 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:47.236938 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:47.238104 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:47.238105 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:47.238267 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:47.240091 (+ 1824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:47.240110 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:47.242403 (+ 2293us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:47.252048 (+ 9645us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:47.252059 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:47.252062 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:47.252384 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:47.252668 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:47.252669 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:47.252669 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:47.252795 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:47.253278 (+ 483us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:47.253285 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:47.253437 (+ 152us) client_negotiation.cc:769] Sending connection context
0926 15:02:47.253538 (+ 101us) client_negotiation.cc:240] Negotiation successful
0926 15:02:47.253715 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":88,"threads_started":1}
I20260926 15:02:47.254659 21572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:47.230263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:44901 (local address 127.8.102.62:36139)
0926 15:02:47.232694 (+ 2431us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:47.232705 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:47.232750 (+ 45us) server_negotiation.cc:407] Connection header received
0926 15:02:47.232905 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:47.232912 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:47.232973 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:47.233110 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:47.233965 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:47.235390 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:47.237080 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:47.237845 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:47.237965 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:47.242598 (+ 4633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:47.242604 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:47.242605 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:47.242621 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:47.251897 (+ 9276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:47.252475 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:47.252478 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:47.252479 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:47.252595 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:47.252859 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:47.252860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:47.252860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:47.253172 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:47.253249 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:47.254277 (+ 1028us) server_negotiation.cc:299] Negotiation successful
0926 15:02:47.254416 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2365,"thread_start_us":95,"threads_started":1}
I20260926 15:02:47.255434 21568 heartbeater.cc:347] Connected to a master server at 127.8.102.62:36139
I20260926 15:02:47.255534 21568 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:47.255733 21568 heartbeater.cc:511] Master 127.8.102.62:36139 requested a full tablet report, sending...
W20260926 15:02:47.256712 21568 heartbeater.cc:644] Failed to heartbeat to 127.8.102.62:36139 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 15:02:47.256786 21568 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:47.256850 21568 heartbeater.cc:511] Master 127.8.102.62:36139 requested a full tablet report, sending...
I20260926 15:02:47.257265 21568 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:47.257342 21568 heartbeater.cc:511] Master 127.8.102.62:36139 requested a full tablet report, sending...
W20260926 15:02:47.257685 21568 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 15:02:47.348076 21570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:47.348341 21570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:47.348379 21570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:47.349757 21570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:47.349838 21570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:47.349884 21570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:02:47.351944 21570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.102.20:59047/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:37313
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=42003
--tserver_master_addrs=127.8.102.62:36139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21570
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:47.352260 21570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:47.352519 21570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:47.358095 21578 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:47.358417 21577 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:47.358482 21570 server_base.cc:1034] running on GCE node
W20260926 15:02:47.358788 21580 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:47.359033 21570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:47.359344 21570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:47.360555 21570 hybrid_clock.cc:648] HybridClock initialized: now 1790434967360525 us; error 47 us; skew 500 ppm
Sep 26 15:02:47 dist-test-slave-jnhs krb5kdc[20374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434967, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:47.367161 21570 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:02:47.367857 21570 webserver.cc:460] Webserver started at http://127.8.102.2:42003/ using document root <none> and password file <none>
I20260926 15:02:47.368269 21570 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:47.368409 21570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 15:02:47 dist-test-slave-jnhs krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434967, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for HTTP/127.8.102.20@KRBTEST.COM
I20260926 15:02:47.389483 21570 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.000s
I20260926 15:02:47.390612 21588 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:47.390941 21570 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:47.391054 21570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "86a3206390894b388647e4673be4d2be"
format_stamp: "Formatted at 2026-09-26 15:02:46 on dist-test-slave-jnhs"
server_key: "a2d9652d21e8a59c6e40954203454091"
server_key_iv: "d636dbf70a4d09dd0e4963d5db0a5ff4"
server_key_version: "kuduclusterkey@0"
I20260926 15:02:47.391559 21570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:47.418777 21570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:47.421126 21570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:47.421312 21570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:47.421584 21570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:02:47.421984 21570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:02:47.422049 21570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:47.422094 21570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:02:47.422122 21570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:47.429502 21570 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:37313
I20260926 15:02:47.429720 21701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:37313 every 8 connection(s)
I20260926 15:02:47.430159 21570 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
Sep 26 15:02:47 dist-test-slave-jnhs krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434967, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:02:47.439122 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 21570
I20260926 15:02:47.439422 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.8.102.20:59047/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:42633
--local_ip_for_outbound_sockets=127.8.102.3
--tserver_master_addrs=127.8.102.62:36139
--webserver_port=43993
--webserver_interface=127.8.102.3
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.8.102.20:59047/kms with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:02:47.442404 21572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:47.431513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:55351 (local address 127.8.102.62:36139)
0926 15:02:47.431562 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:47.431564 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:47.431868 (+ 304us) server_negotiation.cc:407] Connection header received
0926 15:02:47.432545 (+ 677us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:47.432546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:47.432561 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:47.432593 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:47.433554 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:47.434352 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:47.435600 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:47.435825 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:47.435920 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:47.438609 (+ 2689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:47.438611 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:47.438611 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:47.438626 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:47.440538 (+ 1912us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:47.441125 (+ 587us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:47.441127 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:47.441128 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:47.441191 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:47.441548 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:47.441550 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:47.441550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:47.441712 (+ 162us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:47.441791 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:47.442199 (+ 408us) server_negotiation.cc:299] Negotiation successful
0926 15:02:47.442301 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 15:02:47.443231 21704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:47.431191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36139 (local address 127.8.102.2:55351)
0926 15:02:47.431736 (+ 545us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:47.431783 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:47.432418 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:47.432797 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:47.432801 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:47.433048 (+ 247us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:47.433350 (+ 302us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:47.433358 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:47.434480 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:47.434481 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:47.435483 (+ 1002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:47.435486 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:47.435973 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:47.435974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:47.436108 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:47.436727 (+ 619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:47.436737 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:47.438467 (+ 1730us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:47.440676 (+ 2209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:47.440685 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:47.440689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:47.441023 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:47.441265 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:47.441265 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:47.441266 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:47.441402 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:47.441823 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:47.441831 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:47.442012 (+ 181us) client_negotiation.cc:769] Sending connection context
0926 15:02:47.442778 (+ 766us) client_negotiation.cc:240] Negotiation successful
0926 15:02:47.442972 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":86,"threads_started":1}
I20260926 15:02:47.443939 21702 heartbeater.cc:347] Connected to a master server at 127.8.102.62:36139
I20260926 15:02:47.444037 21702 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:47.444247 21702 heartbeater.cc:511] Master 127.8.102.62:36139 requested a full tablet report, sending...
W20260926 15:02:47.444940 21702 heartbeater.cc:644] Failed to heartbeat to 127.8.102.62:36139 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 15:02:47.445017 21702 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:47.445092 21702 heartbeater.cc:511] Master 127.8.102.62:36139 requested a full tablet report, sending...
I20260926 15:02:47.445425 21702 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:47.445493 21702 heartbeater.cc:511] Master 127.8.102.62:36139 requested a full tablet report, sending...
W20260926 15:02:47.445832 21702 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 15:02:47.569300 21705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:47.569630 21705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:47.569676 21705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:47.571651 21705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:47.571723 21705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:47.571776 21705 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:02:47.574375 21705 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.8.102.20:59047/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:42633
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=43993
--tserver_master_addrs=127.8.102.62:36139
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:47.574694 21705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:47.574965 21705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:47.581681 21711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:47.581692 21710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:47.582027 21713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:47.581959 21705 server_base.cc:1034] running on GCE node
I20260926 15:02:47.582329 21705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:47.582654 21705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:47.583837 21705 hybrid_clock.cc:648] HybridClock initialized: now 1790434967583815 us; error 35 us; skew 500 ppm
Sep 26 15:02:47 dist-test-slave-jnhs krb5kdc[20374](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434967, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:47.587419 21705 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:02:47.588016 21705 webserver.cc:460] Webserver started at http://127.8.102.3:43993/ using document root <none> and password file <none>
I20260926 15:02:47.588307 21705 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:47.588392 21705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 26 15:02:47 dist-test-slave-jnhs krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434967, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for HTTP/127.8.102.20@KRBTEST.COM
I20260926 15:02:47.623181 21705 fs_manager.cc:709] Time spent opening directory manager: real 0.016s user 0.001s sys 0.000s
I20260926 15:02:47.624408 21722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:47.624586 21705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:47.624686 21705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "8dd7eb3419bc4dbd86fdcb1ac83ba6d0"
format_stamp: "Formatted at 2026-09-26 15:02:46 on dist-test-slave-jnhs"
server_key: "dee779b93c8a977552fe08004af36ebb"
server_key_iv: "1118027c2d61fc1d54c07d2c91fc67ac"
server_key_version: "kuduclusterkey@0"
I20260926 15:02:47.625207 21705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:47.644830 21705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:47.647430 21705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:47.647607 21705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:47.647887 21705 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:02:47.648272 21705 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:02:47.648322 21705 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:47.648352 21705 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:02:47.648380 21705 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:47.656141 21705 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:42633
I20260926 15:02:47.656669 21705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:02:47.658262 21835 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:42633 every 8 connection(s)
I20260926 15:02:47.661393 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 21705
Sep 26 15:02:47 dist-test-slave-jnhs krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434967, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:02:47.670080 21838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:47.658958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36139 (local address 127.8.102.3:58525)
0926 15:02:47.659545 (+ 587us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:47.659596 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:47.660308 (+ 712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:47.660660 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:47.660664 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:47.660894 (+ 230us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:47.661246 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:47.661254 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:47.662279 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:47.662280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:47.663591 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:47.663595 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:47.664175 (+ 580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:47.664176 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:47.664326 (+ 150us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:47.664888 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:47.664899 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:47.666497 (+ 1598us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:47.668520 (+ 2023us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:47.668527 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:47.668530 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:47.668832 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:47.669023 (+ 191us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:47.669024 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:47.669024 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:47.669138 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:47.669402 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:47.669408 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:47.669573 (+ 165us) client_negotiation.cc:769] Sending connection context
0926 15:02:47.669688 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 15:02:47.669841 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":104,"threads_started":1}
I20260926 15:02:47.670609 21572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:47.659272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:58525 (local address 127.8.102.62:36139)
0926 15:02:47.659321 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:47.659323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:47.659688 (+ 365us) server_negotiation.cc:407] Connection header received
0926 15:02:47.660452 (+ 764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:47.660454 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:47.660467 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:47.660557 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:47.661371 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:47.662111 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:47.663689 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:47.664010 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:47.664083 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:47.666668 (+ 2585us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:47.666669 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:47.666670 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:47.666677 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:47.668423 (+ 1746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:47.668909 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:47.668910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:47.668911 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:47.668964 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:47.669201 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:47.669202 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:47.669202 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:47.669326 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:47.669380 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:47.670409 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0926 15:02:47.670507 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 15:02:47.671321 21836 heartbeater.cc:347] Connected to a master server at 127.8.102.62:36139
I20260926 15:02:47.671422 21836 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:47.671634 21836 heartbeater.cc:511] Master 127.8.102.62:36139 requested a full tablet report, sending...
I20260926 15:02:47.672214 21840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:47.662265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40288 (local address 127.8.102.62:36139)
0926 15:02:47.663926 (+ 1661us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:47.663927 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:47.663935 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:02:47.663956 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:47.663957 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:47.663981 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:47.664062 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:47.664841 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:47.665574 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:47.666688 (+ 1114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:47.666971 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:47.667038 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:47.668751 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:47.668752 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:47.668753 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:47.668759 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:47.670524 (+ 1765us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:47.670993 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:47.670994 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:47.670994 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:47.671069 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:47.671272 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:47.671272 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:47.671273 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:47.671393 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:47.671561 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:47.672011 (+ 450us) server_negotiation.cc:299] Negotiation successful
0926 15:02:47.672114 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1637,"thread_start_us":60,"threads_started":1}
W20260926 15:02:47.672326 21836 heartbeater.cc:644] Failed to heartbeat to 127.8.102.62:36139 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260926 15:02:47.672392 21836 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:47.672451 21836 heartbeater.cc:511] Master 127.8.102.62:36139 requested a full tablet report, sending...
I20260926 15:02:47.673050 21836 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:47.673127 21836 heartbeater.cc:511] Master 127.8.102.62:36139 requested a full tablet report, sending...
W20260926 15:02:47.673545 21836 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Sep 26 15:02:48 dist-test-slave-jnhs krb5kdc[20374](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790434968, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:48.257884 21568 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:48.258030 21568 heartbeater.cc:511] Master 127.8.102.62:36139 requested a full tablet report, sending...
I20260926 15:02:48.258559 21370 ts_manager.cc:194] Registered new tserver with Master: 3b48aa638ca74d34b25ec2d0c008bfc1 (127.8.102.1:33757)
I20260926 15:02:48.259620 21370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:44901
I20260926 15:02:48.446033 21702 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:48.446197 21702 heartbeater.cc:511] Master 127.8.102.62:36139 requested a full tablet report, sending...
I20260926 15:02:48.446722 21370 ts_manager.cc:194] Registered new tserver with Master: 86a3206390894b388647e4673be4d2be (127.8.102.2:37313)
I20260926 15:02:48.447253 21370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:55351
Sep 26 15:02:48 dist-test-slave-jnhs krb5kdc[20374](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434968, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for HTTP/127.8.102.20@KRBTEST.COM
I20260926 15:02:48.673731 21836 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:48.673902 21836 heartbeater.cc:511] Master 127.8.102.62:36139 requested a full tablet report, sending...
I20260926 15:02:48.674360 21370 ts_manager.cc:194] Registered new tserver with Master: 8dd7eb3419bc4dbd86fdcb1ac83ba6d0 (127.8.102.3:42633)
I20260926 15:02:48.674969 21370 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:58525
I20260926 15:02:48.675707 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:02:48.685073 21857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:48.676707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40292 (local address 127.8.102.62:36139)
0926 15:02:48.676990 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:48.676992 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:48.677172 (+ 180us) server_negotiation.cc:407] Connection header received
0926 15:02:48.677198 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:48.677199 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:48.677210 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:48.677272 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:48.678024 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.678867 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.679923 (+ 1056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.680219 (+ 296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.680291 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:48.681871 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:48.681872 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:48.681872 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:48.681880 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:48.683723 (+ 1843us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:48.684153 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:48.684154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:48.684155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:48.684215 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:48.684392 (+ 177us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:48.684393 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:48.684393 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:48.684522 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:48.684612 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:48.684827 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 15:02:48.684936 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":262,"thread_start_us":66,"threads_started":1}
I20260926 15:02:48.689021 21370 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:02:48.691532 21370 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:02:48.711198 21865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:48.695113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:42633 (local address 127.0.0.1:55896)
0926 15:02:48.702798 (+ 7685us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:48.702885 (+ 87us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:48.702966 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:48.703757 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:48.703761 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:48.703763 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:02:48.703855 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:48.703858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.707631 (+ 3773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.707632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:48.708839 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:48.708842 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.710841 (+ 1999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.710842 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:48.710932 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:48.710950 (+ 18us) client_negotiation.cc:769] Sending connection context
0926 15:02:48.711025 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 15:02:48.711090 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7637,"thread_start_us":73,"threads_started":1}
I20260926 15:02:48.712984 21868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:48.695836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:33757 (local address 127.0.0.1:41972)
0926 15:02:48.703305 (+ 7469us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:48.703315 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:48.703358 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:48.704220 (+ 862us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:48.704221 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:48.704223 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:02:48.704262 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:48.704263 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.705904 (+ 1641us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.705905 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:48.709460 (+ 3555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:48.709463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.712714 (+ 3251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.712716 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:48.712786 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:48.712791 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:02:48.712844 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 15:02:48.712891 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7455,"thread_start_us":73,"threads_started":1}
I20260926 15:02:48.715505 21867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:48.695630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55896 (local address 127.8.102.3:42633)
0926 15:02:48.703462 (+ 7832us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:48.703467 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:48.703478 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:02:48.703516 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:48.703519 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:48.703543 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:48.703623 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:02:48.704345 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.707526 (+ 3181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.709814 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.710624 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.710734 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:48.710802 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:48.715313 (+ 4511us) server_negotiation.cc:299] Negotiation successful
0926 15:02:48.715392 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7797,"thread_start_us":61,"threads_started":1}
I20260926 15:02:48.715966 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:48.696094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41972 (local address 127.8.102.1:33757)
0926 15:02:48.703253 (+ 7159us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:48.703259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:48.703987 (+ 728us) server_negotiation.cc:407] Connection header received
0926 15:02:48.704028 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:48.704030 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:48.704047 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:48.704124 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:02:48.704672 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.705803 (+ 1131us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.711701 (+ 5898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.712523 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.712599 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:48.712675 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:48.715796 (+ 3121us) server_negotiation.cc:299] Negotiation successful
0926 15:02:48.715864 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7118,"thread_start_us":46,"threads_started":1}
I20260926 15:02:48.716974 21770 tablet_service.cc:1467] Processing CreateTablet for tablet 1c036b5ebd624e4cb758bc83a16ceb31 (DEFAULT_TABLE table=test-table [id=606f22f74c2d45ce8904bb72653f15c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:02:48.717391 21770 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c036b5ebd624e4cb758bc83a16ceb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:02:48.719507 21864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:48.695357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:37313 (local address 127.0.0.1:40230)
0926 15:02:48.703092 (+ 7735us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:48.703102 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:48.703136 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:48.713201 (+ 10065us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:48.713202 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:48.713205 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:02:48.713250 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:48.713252 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.716736 (+ 3484us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.716737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:48.718055 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:48.718059 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.719240 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.719242 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:48.719318 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:48.719324 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 15:02:48.719359 (+ 35us) client_negotiation.cc:240] Negotiation successful
0926 15:02:48.719412 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7718,"thread_start_us":93,"threads_started":1}
I20260926 15:02:48.719954 21866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:48.695454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40230 (local address 127.8.102.2:37313)
0926 15:02:48.709616 (+ 14162us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:48.709622 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:48.709633 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:02:48.709685 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:48.709687 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:48.709706 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:48.713340 (+ 3634us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:02:48.713440 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.714822 (+ 1382us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.718193 (+ 3371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.719112 (+ 919us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.719708 (+ 596us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:48.719791 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:48.719823 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 15:02:48.719870 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14117,"thread_start_us":66,"threads_started":1}
I20260926 15:02:48.720670 21631 tablet_service.cc:1467] Processing CreateTablet for tablet 1c036b5ebd624e4cb758bc83a16ceb31 (DEFAULT_TABLE table=test-table [id=606f22f74c2d45ce8904bb72653f15c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:02:48.721079 21631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c036b5ebd624e4cb758bc83a16ceb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:02:48.721177 21870 tablet_bootstrap.cc:492] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0: Bootstrap starting.
I20260926 15:02:48.721532 21502 tablet_service.cc:1467] Processing CreateTablet for tablet 1c036b5ebd624e4cb758bc83a16ceb31 (DEFAULT_TABLE table=test-table [id=606f22f74c2d45ce8904bb72653f15c3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:02:48.721844 21502 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1c036b5ebd624e4cb758bc83a16ceb31. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:02:48.730095 21872 tablet_bootstrap.cc:492] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be: Bootstrap starting.
I20260926 15:02:48.731148 21870 tablet_bootstrap.cc:654] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0: Neither blocks nor log segments found. Creating new log.
I20260926 15:02:48.731153 21872 tablet_bootstrap.cc:654] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be: Neither blocks nor log segments found. Creating new log.
I20260926 15:02:48.731529 21870 log.cc:824] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0: Log is configured to *not* fsync() on all Append() calls
I20260926 15:02:48.733206 21871 tablet_bootstrap.cc:492] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1: Bootstrap starting.
I20260926 15:02:48.733719 21872 log.cc:824] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be: Log is configured to *not* fsync() on all Append() calls
I20260926 15:02:48.734625 21871 tablet_bootstrap.cc:654] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1: Neither blocks nor log segments found. Creating new log.
I20260926 15:02:48.739518 21870 tablet_bootstrap.cc:492] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0: No bootstrap required, opened a new log
I20260926 15:02:48.739643 21870 ts_tablet_manager.cc:1397] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20260926 15:02:48.741503 21872 tablet_bootstrap.cc:492] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be: No bootstrap required, opened a new log
I20260926 15:02:48.742020 21871 log.cc:824] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1: Log is configured to *not* fsync() on all Append() calls
I20260926 15:02:48.742404 21872 ts_tablet_manager.cc:1397] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.002s
I20260926 15:02:48.743702 21870 raft_consensus.cc:348] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd7eb3419bc4dbd86fdcb1ac83ba6d0" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42633 } } peers { permanent_uuid: "86a3206390894b388647e4673be4d2be" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37313 } } peers { permanent_uuid: "3b48aa638ca74d34b25ec2d0c008bfc1" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33757 } }
I20260926 15:02:48.743928 21870 raft_consensus.cc:374] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:02:48.743966 21870 raft_consensus.cc:729] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8dd7eb3419bc4dbd86fdcb1ac83ba6d0, State: Initialized, Role: FOLLOWER
I20260926 15:02:48.744144 21870 consensus_queue.cc:260] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd7eb3419bc4dbd86fdcb1ac83ba6d0" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42633 } } peers { permanent_uuid: "86a3206390894b388647e4673be4d2be" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37313 } } peers { permanent_uuid: "3b48aa638ca74d34b25ec2d0c008bfc1" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33757 } }
I20260926 15:02:48.744450 21870 ts_tablet_manager.cc:1428] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260926 15:02:48.744604 21836 heartbeater.cc:503] Master 127.8.102.62:36139 was elected leader, sending a full tablet report...
I20260926 15:02:48.745033 21871 tablet_bootstrap.cc:492] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1: No bootstrap required, opened a new log
I20260926 15:02:48.745111 21871 ts_tablet_manager.cc:1397] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.000s
I20260926 15:02:48.745926 21872 raft_consensus.cc:348] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd7eb3419bc4dbd86fdcb1ac83ba6d0" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42633 } } peers { permanent_uuid: "86a3206390894b388647e4673be4d2be" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37313 } } peers { permanent_uuid: "3b48aa638ca74d34b25ec2d0c008bfc1" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33757 } }
I20260926 15:02:48.746134 21872 raft_consensus.cc:374] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:02:48.746225 21872 raft_consensus.cc:729] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 86a3206390894b388647e4673be4d2be, State: Initialized, Role: FOLLOWER
I20260926 15:02:48.746380 21872 consensus_queue.cc:260] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd7eb3419bc4dbd86fdcb1ac83ba6d0" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42633 } } peers { permanent_uuid: "86a3206390894b388647e4673be4d2be" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37313 } } peers { permanent_uuid: "3b48aa638ca74d34b25ec2d0c008bfc1" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33757 } }
I20260926 15:02:48.746716 21872 ts_tablet_manager.cc:1428] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be: Time spent starting tablet: real 0.004s user 0.003s sys 0.001s
I20260926 15:02:48.746793 21702 heartbeater.cc:503] Master 127.8.102.62:36139 was elected leader, sending a full tablet report...
I20260926 15:02:48.748122 21871 raft_consensus.cc:348] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd7eb3419bc4dbd86fdcb1ac83ba6d0" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42633 } } peers { permanent_uuid: "86a3206390894b388647e4673be4d2be" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37313 } } peers { permanent_uuid: "3b48aa638ca74d34b25ec2d0c008bfc1" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33757 } }
I20260926 15:02:48.748263 21871 raft_consensus.cc:374] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:02:48.748287 21871 raft_consensus.cc:729] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b48aa638ca74d34b25ec2d0c008bfc1, State: Initialized, Role: FOLLOWER
I20260926 15:02:48.748440 21871 consensus_queue.cc:260] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd7eb3419bc4dbd86fdcb1ac83ba6d0" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42633 } } peers { permanent_uuid: "86a3206390894b388647e4673be4d2be" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37313 } } peers { permanent_uuid: "3b48aa638ca74d34b25ec2d0c008bfc1" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33757 } }
I20260926 15:02:48.748740 21871 ts_tablet_manager.cc:1428] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260926 15:02:48.748812 21568 heartbeater.cc:503] Master 127.8.102.62:36139 was elected leader, sending a full tablet report...
I20260926 15:02:48.839517 21878 raft_consensus.cc:482] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:02:48.839696 21878 raft_consensus.cc:504] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd7eb3419bc4dbd86fdcb1ac83ba6d0" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42633 } } peers { permanent_uuid: "86a3206390894b388647e4673be4d2be" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37313 } } peers { permanent_uuid: "3b48aa638ca74d34b25ec2d0c008bfc1" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33757 } }
I20260926 15:02:48.840176 21878 leader_election.cc:288] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8dd7eb3419bc4dbd86fdcb1ac83ba6d0 (127.8.102.3:42633), 86a3206390894b388647e4673be4d2be (127.8.102.2:37313)
I20260926 15:02:48.844640 21867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:48.840702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:60183 (local address 127.8.102.3:42633)
0926 15:02:48.840779 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:48.840781 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:48.840860 (+ 79us) server_negotiation.cc:407] Connection header received
0926 15:02:48.840970 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:48.840971 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:48.840986 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:48.841017 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:02:48.841274 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.842041 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.843580 (+ 1539us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.844211 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.844266 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:48.844280 (+ 14us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:48.844514 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 15:02:48.844560 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:02:48.844878 21866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:48.840678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:59527 (local address 127.8.102.2:37313)
0926 15:02:48.840729 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:48.840732 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:48.840872 (+ 140us) server_negotiation.cc:407] Connection header received
0926 15:02:48.840987 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:48.840988 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:48.841001 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:48.841032 (+ 31us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:02:48.841287 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.842070 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.843821 (+ 1751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.844432 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.844487 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:48.844498 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:48.844756 (+ 258us) server_negotiation.cc:299] Negotiation successful
0926 15:02:48.844803 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 15:02:48.845034 21880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:48.840375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:42633 (local address 127.8.102.1:60183)
0926 15:02:48.840783 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:48.840796 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:48.840890 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:48.841093 (+ 203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:48.841094 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:48.841095 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:02:48.841186 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:48.841189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.842156 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.842157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:48.843460 (+ 1303us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:48.843463 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.844361 (+ 898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.844361 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:48.844430 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:48.844434 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:02:48.844882 (+ 448us) client_negotiation.cc:240] Negotiation successful
0926 15:02:48.844934 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":118,"threads_started":1}
I20260926 15:02:48.845433 21790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c036b5ebd624e4cb758bc83a16ceb31" candidate_uuid: "3b48aa638ca74d34b25ec2d0c008bfc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dd7eb3419bc4dbd86fdcb1ac83ba6d0" is_pre_election: true
I20260926 15:02:48.846072 21790 raft_consensus.cc:2454] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b48aa638ca74d34b25ec2d0c008bfc1 in term 0.
I20260926 15:02:48.846379 21879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:48.840472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:37313 (local address 127.8.102.1:59527)
0926 15:02:48.840795 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:48.840806 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:48.840903 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:48.841105 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:48.841106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:48.841108 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:02:48.841193 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:48.841195 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.842217 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.842218 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:48.843702 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:48.843707 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.844577 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.844578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:48.844665 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:48.844668 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:02:48.845164 (+ 496us) client_negotiation.cc:240] Negotiation successful
0926 15:02:48.845229 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":301,"thread_start_us":84,"threads_started":1}
I20260926 15:02:48.846385 21455 leader_election.cc:302] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b48aa638ca74d34b25ec2d0c008bfc1, 8dd7eb3419bc4dbd86fdcb1ac83ba6d0; no voters:
I20260926 15:02:48.846782 21878 raft_consensus.cc:2781] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:02:48.846851 21878 raft_consensus.cc:482] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:02:48.846875 21878 raft_consensus.cc:3037] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:02:48.847019 21656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c036b5ebd624e4cb758bc83a16ceb31" candidate_uuid: "3b48aa638ca74d34b25ec2d0c008bfc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a3206390894b388647e4673be4d2be" is_pre_election: true
I20260926 15:02:48.847235 21656 raft_consensus.cc:2454] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3b48aa638ca74d34b25ec2d0c008bfc1 in term 0.
I20260926 15:02:48.847847 21878 raft_consensus.cc:504] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd7eb3419bc4dbd86fdcb1ac83ba6d0" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42633 } } peers { permanent_uuid: "86a3206390894b388647e4673be4d2be" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37313 } } peers { permanent_uuid: "3b48aa638ca74d34b25ec2d0c008bfc1" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33757 } }
I20260926 15:02:48.848001 21878 leader_election.cc:288] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1 [CANDIDATE]: Term 1 election: Requested vote from peers 8dd7eb3419bc4dbd86fdcb1ac83ba6d0 (127.8.102.3:42633), 86a3206390894b388647e4673be4d2be (127.8.102.2:37313)
I20260926 15:02:48.848235 21656 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c036b5ebd624e4cb758bc83a16ceb31" candidate_uuid: "3b48aa638ca74d34b25ec2d0c008bfc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "86a3206390894b388647e4673be4d2be"
I20260926 15:02:48.848273 21790 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1c036b5ebd624e4cb758bc83a16ceb31" candidate_uuid: "3b48aa638ca74d34b25ec2d0c008bfc1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8dd7eb3419bc4dbd86fdcb1ac83ba6d0"
I20260926 15:02:48.848312 21656 raft_consensus.cc:3037] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:02:48.848353 21790 raft_consensus.cc:3037] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:02:48.849265 21656 raft_consensus.cc:2454] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b48aa638ca74d34b25ec2d0c008bfc1 in term 1.
I20260926 15:02:48.849269 21790 raft_consensus.cc:2454] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3b48aa638ca74d34b25ec2d0c008bfc1 in term 1.
I20260926 15:02:48.849465 21457 leader_election.cc:302] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b48aa638ca74d34b25ec2d0c008bfc1, 86a3206390894b388647e4673be4d2be; no voters:
I20260926 15:02:48.849664 21878 raft_consensus.cc:2781] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:02:48.849846 21878 raft_consensus.cc:686] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1 [term 1 LEADER]: Becoming Leader. State: Replica: 3b48aa638ca74d34b25ec2d0c008bfc1, State: Running, Role: LEADER
I20260926 15:02:48.849946 21878 consensus_queue.cc:237] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd7eb3419bc4dbd86fdcb1ac83ba6d0" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42633 } } peers { permanent_uuid: "86a3206390894b388647e4673be4d2be" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37313 } } peers { permanent_uuid: "3b48aa638ca74d34b25ec2d0c008bfc1" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33757 } }
I20260926 15:02:48.850860 21369 catalog_manager.cc:5495] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3b48aa638ca74d34b25ec2d0c008bfc1 (127.8.102.1). New cstate: current_term: 1 leader_uuid: "3b48aa638ca74d34b25ec2d0c008bfc1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8dd7eb3419bc4dbd86fdcb1ac83ba6d0" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "86a3206390894b388647e4673be4d2be" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37313 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3b48aa638ca74d34b25ec2d0c008bfc1" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33757 } health_report { overall_health: HEALTHY } } }
I20260926 15:02:48.891757 21869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:48.888526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41984 (local address 127.8.102.1:33757)
0926 15:02:48.888597 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:48.888600 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:48.888609 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:02:48.888651 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:48.888652 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:48.888676 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:48.888757 (+ 81us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:02:48.888925 (+ 168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.889656 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.890791 (+ 1135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:48.891080 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:48.891178 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:48.891360 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:02:48.891501 (+ 141us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:02:48.891602 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:48.891649 (+ 47us) server_negotiation.cc:299] Negotiation successful
0926 15:02:48.891675 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 15:02:48.893942 21656 raft_consensus.cc:1261] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be [term 1 FOLLOWER]: Refusing update from remote peer 3b48aa638ca74d34b25ec2d0c008bfc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:02:48.893978 21790 raft_consensus.cc:1261] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0 [term 1 FOLLOWER]: Refusing update from remote peer 3b48aa638ca74d34b25ec2d0c008bfc1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:02:48.894299 21878 consensus_queue.cc:1035] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8dd7eb3419bc4dbd86fdcb1ac83ba6d0" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 42633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:02:48.894464 21878 consensus_queue.cc:1035] T 1c036b5ebd624e4cb758bc83a16ceb31 P 3b48aa638ca74d34b25ec2d0c008bfc1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "86a3206390894b388647e4673be4d2be" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:02:48.900388 21369 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:02:48.900509 21369 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:02:48.901293 21369 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7d5ddd535b8049c2a7c59d785aa03f02: Sending DeleteTablet for 3 replicas of tablet 1c036b5ebd624e4cb758bc83a16ceb31
I20260926 15:02:48.901672 21631 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c036b5ebd624e4cb758bc83a16ceb31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:02:48 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:40230
I20260926 15:02:48.901840 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 21409
I20260926 15:02:48.902524 21885 mvcc.cc:205] Tried to move back new op lower bound from 7333621632586436608 to 7333621632410046464. Current Snapshot: MvccSnapshot[applied={T|T < 7333621632586436608 or (T in {7333621632586436608})}]
I20260926 15:02:48.902765 21886 mvcc.cc:205] Tried to move back new op lower bound from 7333621632586436608 to 7333621632410046464. Current Snapshot: MvccSnapshot[applied={T|T < 7333621632586436608}]
I20260926 15:02:48.903162 21894 tablet_replica.cc:330] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be: stopping tablet replica
I20260926 15:02:48.903335 21894 raft_consensus.cc:2229] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:02:48.903501 21894 raft_consensus.cc:2258] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:02:48.904038 21894 ts_tablet_manager.cc:1905] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:02:48.905949 21770 tablet_service.cc:1514] Processing DeleteTablet for tablet 1c036b5ebd624e4cb758bc83a16ceb31 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:02:48 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:55896
I20260926 15:02:48.906355 21894 ts_tablet_manager.cc:1918] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:02:48.906482 21894 log.cc:1196] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/1c036b5ebd624e4cb758bc83a16ceb31
I20260926 15:02:48.907080 21894 ts_tablet_manager.cc:1939] T 1c036b5ebd624e4cb758bc83a16ceb31 P 86a3206390894b388647e4673be4d2be: Deleting consensus metadata
I20260926 15:02:48.907605 21356 catalog_manager.cc:4841] TS 86a3206390894b388647e4673be4d2be (127.8.102.2:37313): tablet 1c036b5ebd624e4cb758bc83a16ceb31 (table test-table [id=606f22f74c2d45ce8904bb72653f15c3]) successfully deleted
I20260926 15:02:48.907871 21895 tablet_replica.cc:330] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0: stopping tablet replica
I20260926 15:02:48.908129 21895 raft_consensus.cc:2229] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:02:48.908314 21895 raft_consensus.cc:2258] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:02:48.908845 21895 ts_tablet_manager.cc:1905] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260926 15:02:48.910339 21354 connection.cc:535] server connection from 127.8.102.1:44901 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 15:02:48.910537 21355 connection.cc:535] client connection to 127.8.102.1:33757 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260926 15:02:48.910666 21355 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 15:02:48.910676 21895 ts_tablet_manager.cc:1918] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:02:48.910748 21895 log.cc:1196] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/1c036b5ebd624e4cb758bc83a16ceb31
I20260926 15:02:48.911026 21895 ts_tablet_manager.cc:1939] T 1c036b5ebd624e4cb758bc83a16ceb31 P 8dd7eb3419bc4dbd86fdcb1ac83ba6d0: Deleting consensus metadata
I20260926 15:02:48.911300 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 21570
I20260926 15:02:48.912726 21354 catalog_manager.cc:4841] TS 8dd7eb3419bc4dbd86fdcb1ac83ba6d0 (127.8.102.3:42633): tablet 1c036b5ebd624e4cb758bc83a16ceb31 (table test-table [id=606f22f74c2d45ce8904bb72653f15c3]) successfully deleted
I20260926 15:02:48.914012 21864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:48.913837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:33757 (local address 127.0.0.1:41990)
0926 15:02:48.913902 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:48.913965 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.1:33757: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 15:02:48.914280 21355 catalog_manager.cc:4584] TS 3b48aa638ca74d34b25ec2d0c008bfc1 (127.8.102.1:33757): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1c036b5ebd624e4cb758bc83a16ceb31: Network error: Client connection negotiation failed: client connection to 127.8.102.1:33757: connect: Connection refused (error 111)
I20260926 15:02:48.917690 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 21705
I20260926 15:02:48.924780 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 21337
I20260926 15:02:48.929674 8600 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260926 15:02:49.275187 8600 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260926 15:02:49.275290 8600 mini_ranger.cc:66] Stopping Ranger...
I20260926 15:02:49.620995 8600 mini_ranger.cc:68] Stopped Ranger
2026-09-26 15:02:49.621 UTC [20404] LOG: received smart shutdown request
2026-09-26 15:02:49.626 UTC [20404] LOG: background worker "logical replication launcher" (PID 20413) exited with exit code 1
2026-09-26 15:02:49.626 UTC [20408] LOG: shutting down
2026-09-26 15:02:49.846 UTC [20404] LOG: database system is shut down
2026-09-26T15:02:49Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (51627 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:02:50 dist-test-slave-jnhs krb5kdc[21905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:02:50 dist-test-slave-jnhs krb5kdc[21905](info): set up 2 sockets
Sep 26 15:02:50 dist-test-slave-jnhs krb5kdc[21905](info): commencing operation
krb5kdc: starting...
W20260926 15:02:51.042737 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.065s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:02:51 dist-test-slave-jnhs krb5kdc[21905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:02:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:02:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:02:51.171113 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:43357
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:33475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:43357,127.8.102.61:32827,127.8.102.60:46785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:02:51.254837 21918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:51.255053 21918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:51.255084 21918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:51.256356 21918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:02:51.256403 21918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:51.256418 21918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:51.256431 21918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:02:51.256443 21918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:02:51.257972 21918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:33475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:43357,127.8.102.61:32827,127.8.102.60:46785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:43357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21918
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:51.258239 21918 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:51.258486 21918 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:51.263537 21923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:51.263576 21924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:51.263566 21926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:51.263728 21918 server_base.cc:1034] running on GCE node
I20260926 15:02:51.264070 21918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:51.264334 21918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:51.265563 21918 hybrid_clock.cc:648] HybridClock initialized: now 1790434971265553 us; error 43 us; skew 500 ppm
Sep 26 15:02:51 dist-test-slave-jnhs krb5kdc[21905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434971, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:51.268345 21918 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:02:51.268810 21918 webserver.cc:460] Webserver started at http://127.8.102.62:38233/ using document root <none> and password file <none>
I20260926 15:02:51.268997 21918 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:51.269040 21918 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:02:51.269189 21918 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:02:51.270243 21918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "17639ead8d9944f990918bbe8c84a1bd"
format_stamp: "Formatted at 2026-09-26 15:02:51 on dist-test-slave-jnhs"
server_key: "51770765cae559a26bc1818eae91ce18"
server_key_iv: "6ff1c3e6a63c3456ebdda8dc98f69cf8"
server_key_version: "encryptionkey@0"
I20260926 15:02:51.270552 21918 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "17639ead8d9944f990918bbe8c84a1bd"
format_stamp: "Formatted at 2026-09-26 15:02:51 on dist-test-slave-jnhs"
server_key: "51770765cae559a26bc1818eae91ce18"
server_key_iv: "6ff1c3e6a63c3456ebdda8dc98f69cf8"
server_key_version: "encryptionkey@0"
I20260926 15:02:51.271924 21918 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:02:51.272707 21933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:51.272884 21918 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:02:51.273002 21918 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "17639ead8d9944f990918bbe8c84a1bd"
format_stamp: "Formatted at 2026-09-26 15:02:51 on dist-test-slave-jnhs"
server_key: "51770765cae559a26bc1818eae91ce18"
server_key_iv: "6ff1c3e6a63c3456ebdda8dc98f69cf8"
server_key_version: "encryptionkey@0"
I20260926 15:02:51.273079 21918 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:51.283681 21918 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:51.285996 21918 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:51.286243 21918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:51.290360 21918 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:43357
I20260926 15:02:51.290391 21985 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:43357 every 8 connection(s)
I20260926 15:02:51.290858 21918 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:02:51.291818 21986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:02:51.293467 21986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:02:51.296231 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 21918
I20260926 15:02:51.296331 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:02:51.296551 8600 external_mini_cluster.cc:1428] Setting key 7b5d2d4fe0cf738841ebaba484bbe432
Sep 26 15:02:51 dist-test-slave-jnhs krb5kdc[21905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434971, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.102.61@KRBTEST.COM; defaulting to no policy
I20260926 15:02:51.305167 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.294241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:43357 (local address 127.0.0.1:34696)
0926 15:02:51.294428 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.294468 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:51.295011 (+ 543us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:51.295324 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:51.295326 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:51.295583 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:51.295743 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:51.295747 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.296863 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.296865 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:51.298059 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:51.298063 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.298850 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.298851 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:51.298913 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.299540 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:51.299550 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:51.300970 (+ 1420us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:51.303328 (+ 2358us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:51.303335 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:51.303337 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:51.303598 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:51.303895 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:51.303896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:51.303896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:51.303992 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:51.304508 (+ 516us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:51.304515 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:51.304663 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 15:02:51.304770 (+ 107us) client_negotiation.cc:240] Negotiation successful
0926 15:02:51.304929 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":109,"threads_started":1}
I20260926 15:02:51.305192 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.294255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34696 (local address 127.8.102.62:43357)
0926 15:02:51.294468 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:51.294477 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:51.294553 (+ 76us) server_negotiation.cc:407] Connection header received
0926 15:02:51.295184 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:51.295186 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:51.295213 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:51.295317 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:51.295829 (+ 512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.296770 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.298171 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.298755 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.298881 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.301067 (+ 2186us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:51.301076 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:51.301076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:51.301085 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:51.303213 (+ 2128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.303703 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.303707 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.303708 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.303812 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.304067 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.304068 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.304068 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.304410 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:51.304513 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:51.304830 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 15:02:51.305038 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":89,"server-negotiator.queue_time_us":163,"thread_start_us":131,"threads_started":1}
I20260926 15:02:51.306327 21986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:02:51.306787 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.306620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:32827 (local address 127.0.0.1:39038)
0926 15:02:51.306708 (+ 88us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.306762 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.61:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20260926 15:02:51.306962 21934 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.102.61:32827: connect: Connection refused (error 111)
I20260926 15:02:51.307576 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.307445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:32827 (local address 127.0.0.1:39054)
0926 15:02:51.307514 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.307555 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.61:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260926 15:02:51.307829 21986 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.61:32827: Network error: Client connection negotiation failed: client connection to 127.8.102.61:32827: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.61.keytab.
Entry for principal kudu/127.8.102.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.61.keytab.
I20260926 15:02:51.327279 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.61:32827
--webserver_interface=127.8.102.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.61.keytab
--principal=kudu/127.8.102.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:33475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:43357,127.8.102.61:32827,127.8.102.60:46785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:02:51.357080 21986 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } attempt: 1
I20260926 15:02:51.357657 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.357506 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:32827 (local address 127.0.0.1:39060)
0926 15:02:51.357578 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.357625 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.61:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 15:02:51.358140 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.358047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:32827 (local address 127.0.0.1:39070)
0926 15:02:51.358094 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.358122 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.61:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 15:02:51.358310 21986 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.61:32827: Network error: Client connection negotiation failed: client connection to 127.8.102.61:32827: connect: Connection refused (error 111)
W20260926 15:02:51.411386 21993 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:51.411608 21993 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:51.411633 21993 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:51.412806 21993 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:02:51.412855 21993 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:51.412870 21993 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:51.412883 21993 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:02:51.412895 21993 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:02:51.414474 21993 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:33475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:43357,127.8.102.61:32827,127.8.102.60:46785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.61.keytab
--principal=kudu/127.8.102.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.61:32827
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.102.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21993
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:51.414780 21993 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:51.415036 21993 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:51.420212 21999 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:51.420209 22001 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:51.420231 21998 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:51.420492 21993 server_base.cc:1034] running on GCE node
I20260926 15:02:51.420769 21993 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:51.421100 21993 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:51.422274 21993 hybrid_clock.cc:648] HybridClock initialized: now 1790434971422254 us; error 39 us; skew 500 ppm
Sep 26 15:02:51 dist-test-slave-jnhs krb5kdc[21905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434971, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:51.425019 21993 init.cc:377] Logged in from keytab as kudu/127.8.102.61@KRBTEST.COM (short username kudu)
I20260926 15:02:51.425436 21993 webserver.cc:460] Webserver started at http://127.8.102.61:37233/ using document root <none> and password file <none>
I20260926 15:02:51.425619 21993 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:51.425660 21993 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:02:51.425751 21993 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:02:51.426476 21986 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } attempt: 2
I20260926 15:02:51.426908 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.426771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:32827 (local address 127.0.0.1:39072)
0926 15:02:51.426836 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.426868 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.61:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 15:02:51.426895 21993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-1/data/instance:
uuid: "3db908fccb244f6c8c07f4b0c4ed023c"
format_stamp: "Formatted at 2026-09-26 15:02:51 on dist-test-slave-jnhs"
server_key: "c29b78b1fb9a57eb965e2efa52fea261"
server_key_iv: "bacc0210b0b417949b4ceab30d86e48b"
server_key_version: "encryptionkey@0"
I20260926 15:02:51.427287 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.427208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:32827 (local address 127.0.0.1:39082)
0926 15:02:51.427239 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.427271 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.61:32827: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260926 15:02:51.427376 21986 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.61:32827: Network error: Client connection negotiation failed: client connection to 127.8.102.61:32827: connect: Connection refused (error 111)
I20260926 15:02:51.427376 21993 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-1/wal/instance:
uuid: "3db908fccb244f6c8c07f4b0c4ed023c"
format_stamp: "Formatted at 2026-09-26 15:02:51 on dist-test-slave-jnhs"
server_key: "c29b78b1fb9a57eb965e2efa52fea261"
server_key_iv: "bacc0210b0b417949b4ceab30d86e48b"
server_key_version: "encryptionkey@0"
I20260926 15:02:51.428864 21993 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:51.429780 22008 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:51.430009 21993 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:51.430116 21993 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-1/wal
uuid: "3db908fccb244f6c8c07f4b0c4ed023c"
format_stamp: "Formatted at 2026-09-26 15:02:51 on dist-test-slave-jnhs"
server_key: "c29b78b1fb9a57eb965e2efa52fea261"
server_key_iv: "bacc0210b0b417949b4ceab30d86e48b"
server_key_version: "encryptionkey@0"
I20260926 15:02:51.430181 21993 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:51.440434 21993 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:51.442905 21993 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:51.443032 21993 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:51.446899 21993 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.61:32827
I20260926 15:02:51.446982 22060 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.61:32827 every 8 connection(s)
I20260926 15:02:51.447347 21993 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-1/data/info.pb
I20260926 15:02:51.448437 22061 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:02:51.449991 22061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:02:51.452489 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 21993
I20260926 15:02:51.452651 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-1/wal/instance
I20260926 15:02:51.452862 8600 external_mini_cluster.cc:1428] Setting key e8b1529bd1b07dc1bc7404d078d4884b
Sep 26 15:02:51 dist-test-slave-jnhs krb5kdc[21905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434971, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.61@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:02:51.461293 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.450793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34710 (local address 127.8.102.62:43357)
0926 15:02:51.450865 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:51.450867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:51.451178 (+ 311us) server_negotiation.cc:407] Connection header received
0926 15:02:51.451647 (+ 469us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:51.451648 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:51.451661 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:51.451715 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:51.452374 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.453000 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.454619 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.455050 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.455150 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.458083 (+ 2933us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:51.458085 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:51.458085 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:51.458097 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:51.459709 (+ 1612us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.460259 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.460261 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.460261 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.460318 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.460613 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.460614 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.460614 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.460733 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:51.460786 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:51.461157 (+ 371us) server_negotiation.cc:299] Negotiation successful
0926 15:02:51.461225 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 15:02:51.461580 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.450803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:43357 (local address 127.0.0.1:34710)
0926 15:02:51.450999 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.451080 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:51.451561 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:51.451767 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:51.451770 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:51.452044 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:51.452247 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:51.452251 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.453107 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.453108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:51.454457 (+ 1349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:51.454460 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.455348 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.455348 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:51.455480 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.456126 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:51.456138 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:51.457967 (+ 1829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:51.459815 (+ 1848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:51.459825 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:51.459828 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:51.460172 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:51.460394 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:51.460395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:51.460395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:51.460544 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:51.460805 (+ 261us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:51.460812 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:51.461001 (+ 189us) client_negotiation.cc:769] Sending connection context
0926 15:02:51.461139 (+ 138us) client_negotiation.cc:240] Negotiation successful
0926 15:02:51.461327 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":89,"threads_started":1}
WARNING: no policy specified for kudu/127.8.102.60@KRBTEST.COM; defaulting to no policy
I20260926 15:02:51.462371 22061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.60@KRBTEST.COM" created.
Sep 26 15:02:51 dist-test-slave-jnhs krb5kdc[21905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434971, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.61@KRBTEST.COM for kudu/127.8.102.61@KRBTEST.COM
I20260926 15:02:51.471722 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.462766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:32827 (local address 127.0.0.1:39088)
0926 15:02:51.463064 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.463073 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:51.463110 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:51.463288 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:51.463289 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:51.463411 (+ 122us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:51.463471 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:51.463472 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.464489 (+ 1017us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.464490 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:51.465230 (+ 740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:51.465233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.465801 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.465802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:51.465893 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.466412 (+ 519us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:51.466419 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:51.467855 (+ 1436us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:51.470449 (+ 2594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:51.470450 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:51.470450 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:51.470734 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:51.470993 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:51.470993 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:51.470994 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:51.471054 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:51.471470 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:51.471471 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:51.471511 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 15:02:51.471574 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 15:02:51.471651 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284}
I20260926 15:02:51.472003 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.462939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39088 (local address 127.8.102.61:32827)
0926 15:02:51.463105 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:51.463109 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:51.463129 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:02:51.463167 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:51.463169 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:51.463180 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:51.463286 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:51.463537 (+ 251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.464396 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.465342 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.465695 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.465801 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.467947 (+ 2146us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:51.467958 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:51.467958 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:51.467966 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:51.470328 (+ 2362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.470827 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.470831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.470832 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.470908 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.471143 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.471144 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.471144 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.471385 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:51.471487 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:51.471631 (+ 144us) server_negotiation.cc:299] Negotiation successful
0926 15:02:51.471724 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":83,"threads_started":1}
I20260926 15:02:51.472779 22061 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:02:51.473203 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.473065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:46785 (local address 127.0.0.1:33892)
0926 15:02:51.473119 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.473172 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 15:02:51.473345 22012 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.102.60:46785: connect: Connection refused (error 111)
I20260926 15:02:51.473896 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.473761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:46785 (local address 127.0.0.1:33904)
0926 15:02:51.473828 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.473860 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260926 15:02:51.474025 22061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.60:46785: Network error: Client connection negotiation failed: client connection to 127.8.102.60:46785: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.60.keytab.
Entry for principal kudu/127.8.102.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.60.keytab.
I20260926 15:02:51.486179 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.60:46785
--webserver_interface=127.8.102.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.60.keytab
--principal=kudu/127.8.102.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:33475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:43357,127.8.102.61:32827,127.8.102.60:46785
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:02:51.518575 21986 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } attempt: 3
Sep 26 15:02:51 dist-test-slave-jnhs krb5kdc[21905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434971, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for kudu/127.8.102.61@KRBTEST.COM
I20260926 15:02:51.523275 22061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } attempt: 1
I20260926 15:02:51.523756 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.523604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:46785 (local address 127.0.0.1:33918)
0926 15:02:51.523659 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.523689 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 15:02:51.524287 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.524184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:46785 (local address 127.0.0.1:33920)
0926 15:02:51.524229 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.524256 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 15:02:51.524582 22061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.60:46785: Network error: Client connection negotiation failed: client connection to 127.8.102.60:46785: connect: Connection refused (error 111)
I20260926 15:02:51.526224 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.519015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:32827 (local address 127.0.0.1:39102)
0926 15:02:51.519070 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.519090 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:51.519169 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:51.519338 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:51.519339 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:51.519514 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:51.519582 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:51.519584 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.520367 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.520368 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:51.520900 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:51.520901 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.521310 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.521311 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:51.521397 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.521950 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:51.521955 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:51.523351 (+ 1396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:51.525249 (+ 1898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:51.525250 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:51.525251 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:51.525452 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:51.525670 (+ 218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:51.525670 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:51.525671 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:51.525720 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:51.525974 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:51.525975 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:51.526015 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 15:02:51.526067 (+ 52us) client_negotiation.cc:240] Negotiation successful
0926 15:02:51.526158 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 15:02:51.526268 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.519089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.8.102.61:32827)
0926 15:02:51.519167 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:51.519170 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:51.519181 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:02:51.519245 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:51.519246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:51.519256 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:51.519329 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:51.519684 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.520278 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.520993 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.521219 (+ 226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.521309 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.523457 (+ 2148us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:51.523458 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:51.523459 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:51.523467 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:51.525135 (+ 1668us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.525541 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.525542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.525543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.525607 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.525775 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.525776 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.525776 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.525914 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:51.525988 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:51.526105 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 15:02:51.526189 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:02:51.527118 21986 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:02:51.527552 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.527385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:46785 (local address 127.0.0.1:33930)
0926 15:02:51.527437 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.527464 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 15:02:51.527993 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.527886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:46785 (local address 127.0.0.1:33936)
0926 15:02:51.527939 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.527967 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 15:02:51.528189 21986 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.60:46785: Network error: Client connection negotiation failed: client connection to 127.8.102.60:46785: connect: Connection refused (error 111)
I20260926 15:02:51.559381 21986 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } attempt: 1
I20260926 15:02:51.559882 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.559743 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:46785 (local address 127.0.0.1:33940)
0926 15:02:51.559811 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.559851 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20260926 15:02:51.560266 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.560167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:46785 (local address 127.0.0.1:33942)
0926 15:02:51.560214 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.560237 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 15:02:51.560351 21986 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.60:46785: Network error: Client connection negotiation failed: client connection to 127.8.102.60:46785: connect: Connection refused (error 111)
W20260926 15:02:51.570978 22068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:51.571192 22068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:51.571214 22068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:51.572395 22068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:02:51.572443 22068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:51.572458 22068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:51.572470 22068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:02:51.572482 22068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:02:51.574033 22068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:33475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:43357,127.8.102.61:32827,127.8.102.60:46785
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.60.keytab
--principal=kudu/127.8.102.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.60:46785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.102.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:51.574280 22068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:51.574553 22068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:51.579558 22074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:51.579563 22076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:51.579663 22073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:51.579993 22068 server_base.cc:1034] running on GCE node
I20260926 15:02:51.580228 22068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:51.580561 22068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:51.581733 22068 hybrid_clock.cc:648] HybridClock initialized: now 1790434971581717 us; error 49 us; skew 500 ppm
Sep 26 15:02:51 dist-test-slave-jnhs krb5kdc[21905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434971, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:51.584390 22068 init.cc:377] Logged in from keytab as kudu/127.8.102.60@KRBTEST.COM (short username kudu)
I20260926 15:02:51.584830 22068 webserver.cc:460] Webserver started at http://127.8.102.60:32963/ using document root <none> and password file <none>
I20260926 15:02:51.585036 22068 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:51.585078 22068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:02:51.585172 22068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:02:51.586191 22068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-2/data/instance:
uuid: "47bf20e907c046569d364c0ee1cfb02a"
format_stamp: "Formatted at 2026-09-26 15:02:51 on dist-test-slave-jnhs"
server_key: "8fe0a96d73ae972c17c3eaf6248228ce"
server_key_iv: "357d4cfd76a65a9154e29962b2887523"
server_key_version: "encryptionkey@0"
I20260926 15:02:51.586488 22068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-2/wal/instance:
uuid: "47bf20e907c046569d364c0ee1cfb02a"
format_stamp: "Formatted at 2026-09-26 15:02:51 on dist-test-slave-jnhs"
server_key: "8fe0a96d73ae972c17c3eaf6248228ce"
server_key_iv: "357d4cfd76a65a9154e29962b2887523"
server_key_version: "encryptionkey@0"
I20260926 15:02:51.587870 22068 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:02:51.588603 22083 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:51.588798 22068 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:51.588886 22068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-2/wal
uuid: "47bf20e907c046569d364c0ee1cfb02a"
format_stamp: "Formatted at 2026-09-26 15:02:51 on dist-test-slave-jnhs"
server_key: "8fe0a96d73ae972c17c3eaf6248228ce"
server_key_iv: "357d4cfd76a65a9154e29962b2887523"
server_key_version: "encryptionkey@0"
I20260926 15:02:51.588948 22068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:51.592727 22061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } attempt: 2
I20260926 15:02:51.593164 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.593001 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:46785 (local address 127.0.0.1:33946)
0926 15:02:51.593059 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.593097 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260926 15:02:51.593494 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.593398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:46785 (local address 127.0.0.1:33956)
0926 15:02:51.593442 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.593466 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:46785: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 15:02:51.593581 22061 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.60:46785: Network error: Client connection negotiation failed: client connection to 127.8.102.60:46785: connect: Connection refused (error 111)
I20260926 15:02:51.603760 22068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:51.605826 22068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:51.606000 22068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:51.610215 22068 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.60:46785
I20260926 15:02:51.610270 22135 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.60:46785 every 8 connection(s)
I20260926 15:02:51.610719 22068 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-2/data/info.pb
I20260926 15:02:51.611364 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 22068
I20260926 15:02:51.611503 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790434810362722-8600-0/minicluster-data/master-2/wal/instance
I20260926 15:02:51.611738 8600 external_mini_cluster.cc:1428] Setting key a5ca83475984bd063de9c0dc0ea802e4
I20260926 15:02:51.611815 22136 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:02:51.614228 22136 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:02:51 dist-test-slave-jnhs krb5kdc[21905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.60@KRBTEST.COM
Sep 26 15:02:51 dist-test-slave-jnhs krb5kdc[21905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434971, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.60@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:02:51.623637 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.615037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34718 (local address 127.8.102.62:43357)
0926 15:02:51.615114 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:51.615117 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:51.615367 (+ 250us) server_negotiation.cc:407] Connection header received
0926 15:02:51.615398 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:51.615399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:51.615412 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:51.615480 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:51.615882 (+ 402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.616569 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.617567 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.617896 (+ 329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.617985 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.620351 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:51.620353 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:51.620353 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:51.620362 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:51.622145 (+ 1783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.622579 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.622581 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.622581 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.622674 (+ 93us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.622930 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.622931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.622932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.623121 (+ 189us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:51.623205 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:51.623433 (+ 228us) server_negotiation.cc:299] Negotiation successful
0926 15:02:51.623534 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 15:02:51.623934 22140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.614978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:43357 (local address 127.0.0.1:34718)
0926 15:02:51.615202 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.615244 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:51.615319 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:51.615491 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:51.615493 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:51.615738 (+ 245us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:51.615801 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:51.615802 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.616662 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.616663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:51.617475 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:51.617478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.618008 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.618009 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:51.618075 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.618603 (+ 528us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:51.618613 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:51.620199 (+ 1586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:51.622234 (+ 2035us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:51.622241 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:51.622243 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:51.622504 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:51.622747 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:51.622748 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:51.622748 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:51.622853 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:51.623196 (+ 343us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:51.623202 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:51.623295 (+ 93us) client_negotiation.cc:769] Sending connection context
0926 15:02:51.623390 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 15:02:51.623665 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":191,"mutex_wait_us":106,"thread_start_us":147,"threads_started":1}
I20260926 15:02:51.623936 22139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.612333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33966 (local address 127.8.102.60:46785)
0926 15:02:51.612496 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:51.612506 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:51.612536 (+ 30us) server_negotiation.cc:407] Connection header received
0926 15:02:51.613076 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:51.613085 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:51.613150 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:51.613302 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:51.613931 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.615274 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.616369 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.616920 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.617048 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.619795 (+ 2747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:51.619803 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:51.619803 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:51.619811 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:51.621991 (+ 2180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.622450 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.622453 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.622453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.622570 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.622854 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.622855 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.622855 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.623175 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:51.623323 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:51.623434 (+ 111us) server_negotiation.cc:299] Negotiation successful
0926 15:02:51.623559 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
I20260926 15:02:51.624687 22136 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:02:51 dist-test-slave-jnhs krb5kdc[21905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434971, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.60@KRBTEST.COM for kudu/127.8.102.61@KRBTEST.COM
Sep 26 15:02:51 dist-test-slave-jnhs krb5kdc[21905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:02:51.632846 22140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.625003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:32827 (local address 127.0.0.1:39116)
0926 15:02:51.625048 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.625059 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:51.625103 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:51.625369 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:51.625370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:51.625545 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:51.625583 (+ 38us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:51.625584 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.626289 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.626290 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:51.627295 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:51.627298 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.627772 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.627772 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:51.627822 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.628304 (+ 482us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:51.628309 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:51.629834 (+ 1525us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:51.631609 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:51.631610 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:51.631611 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:51.631810 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:51.632059 (+ 249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:51.632060 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:51.632060 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:51.632138 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:51.632494 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:51.632495 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:51.632547 (+ 52us) client_negotiation.cc:769] Sending connection context
0926 15:02:51.632630 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 15:02:51.632771 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260926 15:02:51.632912 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.625075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39116 (local address 127.8.102.61:32827)
0926 15:02:51.625173 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:51.625176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:51.625187 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:02:51.625236 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:51.625237 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:51.625251 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:51.625328 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:51.625670 (+ 342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.626191 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.627389 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.627689 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.627768 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.629929 (+ 2161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:51.629931 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:51.629931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:51.629939 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:51.631523 (+ 1584us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.631914 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.631915 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.631915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.631969 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.632224 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.632225 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.632225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.632384 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:51.632469 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:51.632710 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 15:02:51.632790 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260926 15:02:51.633606 22136 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:02:51.635080 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.626737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34724 (local address 127.8.102.62:43357)
0926 15:02:51.626794 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:51.626796 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:51.626804 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:02:51.626869 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:51.626870 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:51.626877 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:51.626921 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:51.627562 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.628142 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.629125 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.629320 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.629379 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.631640 (+ 2261us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:51.631641 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:51.631641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:51.631649 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:51.633518 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.633913 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.633915 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.633915 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.633978 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.634233 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.634234 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.634235 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.634352 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:51.634608 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:51.634864 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 15:02:51.634951 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260926 15:02:51.635468 21986 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } attempt: 2
Sep 26 15:02:51 dist-test-slave-jnhs krb5kdc[21905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434971, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.60@KRBTEST.COM for kudu/127.8.102.60@KRBTEST.COM
Sep 26 15:02:51 dist-test-slave-jnhs krb5kdc[21905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434971, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for kudu/127.8.102.60@KRBTEST.COM
Sep 26 15:02:51 dist-test-slave-jnhs krb5kdc[21905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.61@KRBTEST.COM
I20260926 15:02:51.641991 22140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.633938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:46785 (local address 127.0.0.1:33974)
0926 15:02:51.633984 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.633995 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:51.634037 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:51.634177 (+ 140us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:51.634178 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:51.634318 (+ 140us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:51.634345 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:51.634346 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.635123 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.635123 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:51.636260 (+ 1137us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:51.636263 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.636822 (+ 559us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.636823 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:51.636877 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.637335 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:51.637340 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:51.638810 (+ 1470us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:51.640882 (+ 2072us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:51.640883 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:51.640884 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:51.641052 (+ 168us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:51.641296 (+ 244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:51.641296 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:51.641297 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:51.641354 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:51.641660 (+ 306us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:51.641661 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:51.641701 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 15:02:51.641771 (+ 70us) client_negotiation.cc:240] Negotiation successful
0926 15:02:51.641873 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 15:02:51.642314 22139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.633979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33974 (local address 127.8.102.60:46785)
0926 15:02:51.634019 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:51.634021 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:51.634030 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:02:51.634098 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:51.634099 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:51.634108 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:51.634168 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:51.634425 (+ 257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.635028 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.636347 (+ 1319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.636761 (+ 414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.636827 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.638893 (+ 2066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:51.638894 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:51.638894 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:51.638902 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:51.640753 (+ 1851us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.641153 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.641154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.641154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.641206 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.641425 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.641427 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.641427 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.641558 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:51.641637 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:51.642119 (+ 482us) server_negotiation.cc:299] Negotiation successful
0926 15:02:51.642193 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 15:02:51.644445 22136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a: Bootstrap starting.
I20260926 15:02:51.644876 21988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.635695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:46785 (local address 127.0.0.1:33980)
0926 15:02:51.635789 (+ 94us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.635802 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:51.635845 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:51.636105 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:51.636106 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:51.636269 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:51.636313 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:51.636315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.637769 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.637770 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:51.638684 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:51.638694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.639219 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.639220 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:51.639289 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.639737 (+ 448us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:51.639742 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:51.641186 (+ 1444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:51.643684 (+ 2498us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:51.643685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:51.643685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:51.643910 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:51.644162 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:51.644163 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:51.644164 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:51.644227 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:51.644587 (+ 360us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:51.644589 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:51.644647 (+ 58us) client_negotiation.cc:769] Sending connection context
0926 15:02:51.644719 (+ 72us) client_negotiation.cc:240] Negotiation successful
0926 15:02:51.644811 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20260926 15:02:51.645268 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.635781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33980 (local address 127.8.102.60:46785)
0926 15:02:51.635948 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:51.635950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:51.635977 (+ 27us) server_negotiation.cc:407] Connection header received
0926 15:02:51.636000 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:51.636000 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:51.636012 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:51.636104 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:51.636414 (+ 310us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.637617 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.638777 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.639131 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.639187 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.641294 (+ 2107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:51.641295 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:51.641296 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:51.641303 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:51.643565 (+ 2262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.644006 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.644007 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.644007 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.644077 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.644308 (+ 231us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.644309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.644310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.644476 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:51.644624 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:51.645038 (+ 414us) server_negotiation.cc:299] Negotiation successful
0926 15:02:51.645123 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":108,"threads_started":1}
I20260926 15:02:51.645360 22136 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a: Neither blocks nor log segments found. Creating new log.
I20260926 15:02:51.645730 22136 log.cc:824] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a: Log is configured to *not* fsync() on all Append() calls
I20260926 15:02:51.645874 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.635736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39132 (local address 127.8.102.61:32827)
0926 15:02:51.635773 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:51.635775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:51.635981 (+ 206us) server_negotiation.cc:407] Connection header received
0926 15:02:51.636012 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:51.636013 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:51.636024 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:51.636106 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:51.636781 (+ 675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.637595 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.638375 (+ 780us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.638713 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.638780 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.642029 (+ 3249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:51.642031 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:51.642031 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:51.642039 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:51.643942 (+ 1903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.644409 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.644410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.644410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.644468 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.644706 (+ 238us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.644707 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.644707 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.644862 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:51.645642 (+ 780us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:51.645694 (+ 52us) server_negotiation.cc:299] Negotiation successful
0926 15:02:51.645781 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 15:02:51.647382 22136 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a: No bootstrap required, opened a new log
I20260926 15:02:51.647395 21986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd: Bootstrap starting.
I20260926 15:02:51.647548 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:02:51.648169 21986 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd: Neither blocks nor log segments found. Creating new log.
I20260926 15:02:51.648473 21986 log.cc:824] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd: Log is configured to *not* fsync() on all Append() calls
I20260926 15:02:51.649243 22136 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } } peers { permanent_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } } peers { permanent_uuid: "47bf20e907c046569d364c0ee1cfb02a" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } }
I20260926 15:02:51.649539 22136 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:02:51.649590 22136 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47bf20e907c046569d364c0ee1cfb02a, State: Initialized, Role: FOLLOWER
I20260926 15:02:51.649761 22136 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } } peers { permanent_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } } peers { permanent_uuid: "47bf20e907c046569d364c0ee1cfb02a" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } }
I20260926 15:02:51.650104 21986 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd: No bootstrap required, opened a new log
I20260926 15:02:51.650106 22136 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:02:51.652661 21986 raft_consensus.cc:348] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } } peers { permanent_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } } peers { permanent_uuid: "47bf20e907c046569d364c0ee1cfb02a" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } }
I20260926 15:02:51.652801 21986 raft_consensus.cc:374] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:02:51.652846 21986 raft_consensus.cc:729] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17639ead8d9944f990918bbe8c84a1bd, State: Initialized, Role: FOLLOWER
I20260926 15:02:51.653100 21986 consensus_queue.cc:260] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } } peers { permanent_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } } peers { permanent_uuid: "47bf20e907c046569d364c0ee1cfb02a" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } }
I20260926 15:02:51.653442 21986 sys_catalog.cc:564] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:02:51.653519 22163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } } peers { permanent_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } } peers { permanent_uuid: "47bf20e907c046569d364c0ee1cfb02a" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } } }
I20260926 15:02:51.653606 22163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:02:51.655191 22151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } } peers { permanent_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } } peers { permanent_uuid: "47bf20e907c046569d364c0ee1cfb02a" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } } }
I20260926 15:02:51.655300 22151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a [sys.catalog]: This master's current role is: FOLLOWER
W20260926 15:02:51.656139 22162 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:02:51.656200 22162 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:02:51.657804 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.649109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33982 (local address 127.8.102.60:46785)
0926 15:02:51.649148 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:51.649151 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:51.649628 (+ 477us) server_negotiation.cc:407] Connection header received
0926 15:02:51.649722 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:51.649723 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:51.649734 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:51.649763 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:51.650439 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.651266 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.652457 (+ 1191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.652671 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.652756 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.654702 (+ 1946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:51.654703 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:51.654704 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:51.654712 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:51.656394 (+ 1682us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.656859 (+ 465us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.656860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.656860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.656924 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.657126 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.657127 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.657127 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.657260 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:51.657383 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:51.657615 (+ 232us) server_negotiation.cc:299] Negotiation successful
0926 15:02:51.657708 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 15:02:51.653725 22068 master_runner.cc:186] Error getting master registration for 127.8.102.62:43357: OK, instance_id { permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" instance_seqno: 1790434971289194 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 15:02:51.658071 22068 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 15:02:51.656347 21918 master_runner.cc:186] Error getting master registration for 127.8.102.61:32827: OK, instance_id { permanent_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" instance_seqno: 1790434971445741 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 15:02:51.659448 21918 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 15:02:51.659961 22174 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:02:51.660006 22174 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:02:51.660058 22065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.649252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39140 (local address 127.8.102.61:32827)
0926 15:02:51.649342 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:51.649344 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:51.649357 (+ 13us) server_negotiation.cc:407] Connection header received
0926 15:02:51.649438 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:51.649439 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:51.649449 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:51.649498 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:51.650516 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.651265 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.652473 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.652781 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.652858 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.654797 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:51.654798 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:51.654799 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:51.654805 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:51.657479 (+ 2674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.659061 (+ 1582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.659062 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.659062 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.659128 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.659349 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.659350 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.659350 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.659479 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:51.659563 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:51.659844 (+ 281us) server_negotiation.cc:299] Negotiation successful
0926 15:02:51.659951 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 15:02:51.664754 21989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.649428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34740 (local address 127.8.102.62:43357)
0926 15:02:51.649473 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:51.649476 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:51.655061 (+ 5585us) server_negotiation.cc:407] Connection header received
0926 15:02:51.655086 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:51.655088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:51.655098 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:51.655145 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:51.658080 (+ 2935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.658864 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.659799 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.660066 (+ 267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.660108 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.661754 (+ 1646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:51.661756 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:51.661756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:51.661763 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:51.663441 (+ 1678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.663865 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.663867 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.663867 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.663925 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.664124 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.664125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.664125 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.664258 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:51.664341 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:51.664572 (+ 231us) server_negotiation.cc:299] Negotiation successful
0926 15:02:51.664661 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 15:02:51.684753 22061 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } attempt: 3
Sep 26 15:02:51 dist-test-slave-jnhs krb5kdc[21905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434971, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.61@KRBTEST.COM for kudu/127.8.102.60@KRBTEST.COM
I20260926 15:02:51.693596 22141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.685169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33986 (local address 127.8.102.60:46785)
0926 15:02:51.685228 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:51.685230 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:51.685239 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:02:51.685267 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:51.685268 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:51.685280 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:51.685350 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:51.685708 (+ 358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.686505 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.687526 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.687845 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.687918 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.690536 (+ 2618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:51.690538 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:51.690539 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:51.690547 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:51.692283 (+ 1736us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.692724 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.692725 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.692726 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.692798 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:51.693034 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:51.693035 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:51.693035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:51.693183 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:51.693254 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:51.693423 (+ 169us) server_negotiation.cc:299] Negotiation successful
0926 15:02:51.693506 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 15:02:51.693596 22063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:51.685100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:46785 (local address 127.0.0.1:33986)
0926 15:02:51.685161 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:51.685173 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:51.685211 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:51.685371 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:51.685372 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:51.685566 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:51.685622 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:51.685624 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.686598 (+ 974us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.686599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:51.687411 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:51.687414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:51.688266 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:51.688267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:51.688330 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:51.688866 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:51.688872 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:51.690441 (+ 1569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:51.692391 (+ 1950us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:51.692393 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:51.692393 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:51.692631 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:51.692895 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:51.692896 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:51.692896 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:51.692966 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:51.693258 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:51.693259 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:51.693310 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 15:02:51.693384 (+ 74us) client_negotiation.cc:240] Negotiation successful
0926 15:02:51.693489 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 15:02:51.695745 22061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c: Bootstrap starting.
I20260926 15:02:51.696556 22061 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c: Neither blocks nor log segments found. Creating new log.
I20260926 15:02:51.696888 22061 log.cc:824] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c: Log is configured to *not* fsync() on all Append() calls
I20260926 15:02:51.698518 22061 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c: No bootstrap required, opened a new log
I20260926 15:02:51.700601 22061 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } } peers { permanent_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } } peers { permanent_uuid: "47bf20e907c046569d364c0ee1cfb02a" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } }
I20260926 15:02:51.700773 22061 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:02:51.700809 22061 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3db908fccb244f6c8c07f4b0c4ed023c, State: Initialized, Role: FOLLOWER
I20260926 15:02:51.700958 22061 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } } peers { permanent_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } } peers { permanent_uuid: "47bf20e907c046569d364c0ee1cfb02a" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } }
I20260926 15:02:51.701310 22061 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:02:51.701282 22178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } } peers { permanent_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } } peers { permanent_uuid: "47bf20e907c046569d364c0ee1cfb02a" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } } }
I20260926 15:02:51.701464 22178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [sys.catalog]: This master's current role is: FOLLOWER
W20260926 15:02:51.703189 22189 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:02:51.703303 22189 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:02:51.705492 21993 master_runner.cc:263] No leader master found from master 3db908fccb244f6c8c07f4b0c4ed023c
I20260926 15:02:51.705551 21993 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260926 15:02:51.789402 21939 tablet.cc:2369] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:02:51.856753 22089 tablet.cc:2369] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:02:51.943814 22014 tablet.cc:2369] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:02:51.948738 22178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:02:51.948858 22178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } } peers { permanent_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } } peers { permanent_uuid: "47bf20e907c046569d364c0ee1cfb02a" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } }
I20260926 15:02:51.949093 22178 leader_election.cc:288] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 17639ead8d9944f990918bbe8c84a1bd (127.8.102.62:43357), 47bf20e907c046569d364c0ee1cfb02a (127.8.102.60:46785)
I20260926 15:02:51.949419 21960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17639ead8d9944f990918bbe8c84a1bd" is_pre_election: true
I20260926 15:02:51.949424 22110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47bf20e907c046569d364c0ee1cfb02a" is_pre_election: true
I20260926 15:02:51.949582 21960 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db908fccb244f6c8c07f4b0c4ed023c in term 0.
I20260926 15:02:51.949606 22110 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3db908fccb244f6c8c07f4b0c4ed023c in term 0.
I20260926 15:02:51.949827 22010 leader_election.cc:302] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17639ead8d9944f990918bbe8c84a1bd, 3db908fccb244f6c8c07f4b0c4ed023c; no voters:
I20260926 15:02:51.950024 22178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:02:51.950147 22178 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:02:51.950196 22178 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:02:51.951020 22178 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } } peers { permanent_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } } peers { permanent_uuid: "47bf20e907c046569d364c0ee1cfb02a" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } }
I20260926 15:02:51.951174 22178 leader_election.cc:288] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [CANDIDATE]: Term 1 election: Requested vote from peers 17639ead8d9944f990918bbe8c84a1bd (127.8.102.62:43357), 47bf20e907c046569d364c0ee1cfb02a (127.8.102.60:46785)
I20260926 15:02:51.951385 22110 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47bf20e907c046569d364c0ee1cfb02a"
I20260926 15:02:51.951393 21960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "17639ead8d9944f990918bbe8c84a1bd"
I20260926 15:02:51.951494 22110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:02:51.951494 21960 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:02:51.952443 21960 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db908fccb244f6c8c07f4b0c4ed023c in term 1.
I20260926 15:02:51.952437 22110 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3db908fccb244f6c8c07f4b0c4ed023c in term 1.
I20260926 15:02:51.952687 22010 leader_election.cc:302] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17639ead8d9944f990918bbe8c84a1bd, 3db908fccb244f6c8c07f4b0c4ed023c; no voters:
I20260926 15:02:51.952798 22178 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:02:51.952925 22178 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [term 1 LEADER]: Becoming Leader. State: Replica: 3db908fccb244f6c8c07f4b0c4ed023c, State: Running, Role: LEADER
I20260926 15:02:51.953039 22178 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } } peers { permanent_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } } peers { permanent_uuid: "47bf20e907c046569d364c0ee1cfb02a" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } }
I20260926 15:02:51.953630 22191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3db908fccb244f6c8c07f4b0c4ed023c. Latest consensus state: current_term: 1 leader_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } } peers { permanent_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } } peers { permanent_uuid: "47bf20e907c046569d364c0ee1cfb02a" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } } }
I20260926 15:02:51.953760 22191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [sys.catalog]: This master's current role is: LEADER
I20260926 15:02:51.954093 22193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:02:51.954382 22193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:02:51.956015 21960 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd [term 1 FOLLOWER]: Refusing update from remote peer 3db908fccb244f6c8c07f4b0c4ed023c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:02:51.956142 22110 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a [term 1 FOLLOWER]: Refusing update from remote peer 3db908fccb244f6c8c07f4b0c4ed023c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:02:51.956357 22191 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [LEADER]: Connected to new peer: Peer: permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:02:51.956477 22178 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [LEADER]: Connected to new peer: Peer: permanent_uuid: "47bf20e907c046569d364c0ee1cfb02a" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:02:51.957656 22163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3db908fccb244f6c8c07f4b0c4ed023c. Latest consensus state: current_term: 1 leader_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } } peers { permanent_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } } peers { permanent_uuid: "47bf20e907c046569d364c0ee1cfb02a" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } } }
I20260926 15:02:51.957759 22163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:02:51.957777 22151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3db908fccb244f6c8c07f4b0c4ed023c. Latest consensus state: current_term: 1 leader_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } } peers { permanent_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } } peers { permanent_uuid: "47bf20e907c046569d364c0ee1cfb02a" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } } }
I20260926 15:02:51.957883 22151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:02:51.958688 22194 mvcc.cc:205] Tried to move back new op lower bound from 7333621645129461760 to 7333621645120442368. Current Snapshot: MvccSnapshot[applied={T|T < 7333621645129461760}]
I20260926 15:02:51.958779 22191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } } peers { permanent_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } } peers { permanent_uuid: "47bf20e907c046569d364c0ee1cfb02a" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } } }
I20260926 15:02:51.958868 22191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [sys.catalog]: This master's current role is: LEADER
I20260926 15:02:51.958818 22178 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } } peers { permanent_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } } peers { permanent_uuid: "47bf20e907c046569d364c0ee1cfb02a" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } } }
I20260926 15:02:51.958909 22178 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c [sys.catalog]: This master's current role is: LEADER
I20260926 15:02:51.958971 22195 mvcc.cc:205] Tried to move back new op lower bound from 7333621645129461760 to 7333621645120442368. Current Snapshot: MvccSnapshot[applied={T|T < 7333621645129461760}]
I20260926 15:02:51.959049 22163 sys_catalog.cc:455] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } } peers { permanent_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } } peers { permanent_uuid: "47bf20e907c046569d364c0ee1cfb02a" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } } }
I20260926 15:02:51.959131 22163 sys_catalog.cc:458] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:02:51.959324 22193 catalog_manager.cc:1347] Generated new cluster ID: ceb693ae3ba04e03b9cba272dfc0c5f3
I20260926 15:02:51.959375 22193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:02:51.959772 22151 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "17639ead8d9944f990918bbe8c84a1bd" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43357 } } peers { permanent_uuid: "3db908fccb244f6c8c07f4b0c4ed023c" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 32827 } } peers { permanent_uuid: "47bf20e907c046569d364c0ee1cfb02a" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 46785 } } }
I20260926 15:02:51.959853 22151 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:02:51.970381 22193 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:02:51.971081 22193 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:02:51.980999 22193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3db908fccb244f6c8c07f4b0c4ed023c: Generated new TSK 0
I20260926 15:02:51.981245 22193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:02:52.656672 22162 catalog_manager.cc:1259] Loaded cluster ID: ceb693ae3ba04e03b9cba272dfc0c5f3
I20260926 15:02:52.656775 22162 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a: loading cluster ID for follower catalog manager: success
I20260926 15:02:52.657590 22162 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a: acquiring CA information for follower catalog manager: success
I20260926 15:02:52.657892 22162 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 47bf20e907c046569d364c0ee1cfb02a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 15:02:52.660408 22174 catalog_manager.cc:1259] Loaded cluster ID: ceb693ae3ba04e03b9cba272dfc0c5f3
I20260926 15:02:52.660468 22174 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd: loading cluster ID for follower catalog manager: success
I20260926 15:02:52.661382 22174 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd: acquiring CA information for follower catalog manager: success
I20260926 15:02:52.661713 22174 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 17639ead8d9944f990918bbe8c84a1bd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 15:02:53.054288 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 21918
I20260926 15:02:53.059599 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 21993
I20260926 15:02:53.064865 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 22068
2026-09-26T15:02:53Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (3106 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:02:53 dist-test-slave-jnhs krb5kdc[22209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:02:53 dist-test-slave-jnhs krb5kdc[22209](info): set up 2 sockets
Sep 26 15:02:53 dist-test-slave-jnhs krb5kdc[22209](info): commencing operation
krb5kdc: starting...
W20260926 15:02:54.099004 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.014s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:02:54 dist-test-slave-jnhs krb5kdc[22209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:02:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:02:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:02:54.230306 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:41871
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:41937
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:41871
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:02:54.312436 22222 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:54.312654 22222 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:54.312678 22222 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:54.313793 22222 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:02:54.313835 22222 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:54.313850 22222 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:54.313863 22222 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:02:54.313875 22222 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:02:54.315433 22222 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:41937
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:41871
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:41871
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22222
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:54.315652 22222 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:54.315829 22222 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:54.320887 22230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:54.320905 22227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:54.320955 22228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:54.321026 22222 server_base.cc:1034] running on GCE node
I20260926 15:02:54.321369 22222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:54.321681 22222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:54.322832 22222 hybrid_clock.cc:648] HybridClock initialized: now 1790434974322817 us; error 28 us; skew 500 ppm
Sep 26 15:02:54 dist-test-slave-jnhs krb5kdc[22209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434974, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:54.332564 22222 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:02:54.333215 22222 webserver.cc:460] Webserver started at http://127.8.102.62:46133/ using document root <none> and password file <none>
I20260926 15:02:54.333501 22222 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:54.333576 22222 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:02:54.333726 22222 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:02:54.335403 22222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "b92d576b13fa45c28e804905bb16aae4"
format_stamp: "Formatted at 2026-09-26 15:02:54 on dist-test-slave-jnhs"
server_key: "3a903403d0767dba338b99f66090eb97"
server_key_iv: "4ad3a426608b08bf96ced02c8a304582"
server_key_version: "encryptionkey@0"
I20260926 15:02:54.335901 22222 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "b92d576b13fa45c28e804905bb16aae4"
format_stamp: "Formatted at 2026-09-26 15:02:54 on dist-test-slave-jnhs"
server_key: "3a903403d0767dba338b99f66090eb97"
server_key_iv: "4ad3a426608b08bf96ced02c8a304582"
server_key_version: "encryptionkey@0"
I20260926 15:02:54.338044 22222 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:02:54.339032 22237 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:54.339406 22222 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 15:02:54.339505 22222 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "b92d576b13fa45c28e804905bb16aae4"
format_stamp: "Formatted at 2026-09-26 15:02:54 on dist-test-slave-jnhs"
server_key: "3a903403d0767dba338b99f66090eb97"
server_key_iv: "4ad3a426608b08bf96ced02c8a304582"
server_key_version: "encryptionkey@0"
I20260926 15:02:54.339584 22222 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:54.364351 22222 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:54.367425 22222 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:54.367590 22222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:54.372305 22222 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:41871
I20260926 15:02:54.372432 22289 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:41871 every 8 connection(s)
I20260926 15:02:54.372797 22222 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:02:54.373919 22290 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:02:54.375730 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 22222
I20260926 15:02:54.375845 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:02:54.376056 8600 external_mini_cluster.cc:1428] Setting key 10ba1e29fa5c579019a1b3dc4abac1bd
I20260926 15:02:54.377857 22290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4: Bootstrap starting.
I20260926 15:02:54.378680 22290 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4: Neither blocks nor log segments found. Creating new log.
I20260926 15:02:54.379041 22290 log.cc:824] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4: Log is configured to *not* fsync() on all Append() calls
I20260926 15:02:54.379884 22290 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4: No bootstrap required, opened a new log
I20260926 15:02:54.381994 22290 raft_consensus.cc:348] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92d576b13fa45c28e804905bb16aae4" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41871 } }
I20260926 15:02:54.382133 22290 raft_consensus.cc:374] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:02:54.382156 22290 raft_consensus.cc:729] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b92d576b13fa45c28e804905bb16aae4, State: Initialized, Role: FOLLOWER
I20260926 15:02:54.382302 22290 consensus_queue.cc:260] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92d576b13fa45c28e804905bb16aae4" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41871 } }
I20260926 15:02:54.382382 22290 raft_consensus.cc:388] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:02:54.382440 22290 raft_consensus.cc:482] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:02:54.382491 22290 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:02:54.383106 22290 raft_consensus.cc:504] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92d576b13fa45c28e804905bb16aae4" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41871 } }
I20260926 15:02:54.383236 22290 leader_election.cc:302] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b92d576b13fa45c28e804905bb16aae4; no voters:
Sep 26 15:02:54 dist-test-slave-jnhs krb5kdc[22209](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434974, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:02:54.383452 22290 leader_election.cc:288] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:02:54.383491 22295 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:02:54.383632 22295 raft_consensus.cc:686] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4 [term 1 LEADER]: Becoming Leader. State: Replica: b92d576b13fa45c28e804905bb16aae4, State: Running, Role: LEADER
I20260926 15:02:54.383720 22290 sys_catalog.cc:564] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:02:54.383739 22295 consensus_queue.cc:237] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92d576b13fa45c28e804905bb16aae4" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41871 } }
I20260926 15:02:54.384064 22295 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b92d576b13fa45c28e804905bb16aae4. Latest consensus state: current_term: 1 leader_uuid: "b92d576b13fa45c28e804905bb16aae4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92d576b13fa45c28e804905bb16aae4" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41871 } } }
I20260926 15:02:54.384142 22295 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4 [sys.catalog]: This master's current role is: LEADER
I20260926 15:02:54.384073 22296 sys_catalog.cc:455] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b92d576b13fa45c28e804905bb16aae4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b92d576b13fa45c28e804905bb16aae4" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 41871 } } }
I20260926 15:02:54.384294 22296 sys_catalog.cc:458] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4 [sys.catalog]: This master's current role is: LEADER
I20260926 15:02:54.384573 22301 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:02:54.385070 22301 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:02:54.387832 22301 catalog_manager.cc:1347] Generated new cluster ID: a81095131a5b4bc9aea08672fddae65d
I20260926 15:02:54.387882 22301 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:02:54.388476 22293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:54.376589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58612 (local address 127.8.102.62:41871)
0926 15:02:54.376759 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:54.376775 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:54.376821 (+ 46us) server_negotiation.cc:407] Connection header received
0926 15:02:54.377417 (+ 596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:54.377426 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:54.377507 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:54.377652 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:54.378628 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:54.379884 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:54.380699 (+ 815us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:54.381132 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:54.381252 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:54.384025 (+ 2773us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:54.384029 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:54.384029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:54.384042 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:54.386551 (+ 2509us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:54.387097 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:54.387100 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:54.387101 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:54.387204 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:54.387422 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:54.387423 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:54.387423 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:54.387686 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:54.387845 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:54.388210 (+ 365us) server_negotiation.cc:299] Negotiation successful
0926 15:02:54.388334 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260926 15:02:54.389459 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:02:54.402182 22301 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:02:54.402937 22301 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:02:54.410925 22301 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b92d576b13fa45c28e804905bb16aae4: Generated new TSK 0
I20260926 15:02:54.411130 22301 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:02:54.443615 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 22222
2026-09-26T15:02:54Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1377 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:02:54 dist-test-slave-jnhs krb5kdc[22317](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:02:54 dist-test-slave-jnhs krb5kdc[22317](info): set up 2 sockets
Sep 26 15:02:54 dist-test-slave-jnhs krb5kdc[22317](info): commencing operation
krb5kdc: starting...
W20260926 15:02:55.518941 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.057s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:02:55 dist-test-slave-jnhs krb5kdc[22317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:02:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:02:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:02:55.643762 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:45439
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:45439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:02:55.729243 22330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:55.729481 22330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:55.729506 22330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:55.730700 22330 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:02:55.730739 22330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:55.730753 22330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:55.730767 22330 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:02:55.730778 22330 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:02:55.732244 22330 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:45439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:45439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22330
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:55.732447 22330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:55.732625 22330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:55.738016 22335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:55.738013 22338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:55.738137 22330 server_base.cc:1034] running on GCE node
W20260926 15:02:55.738013 22336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:55.738595 22330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:55.738978 22330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:55.740134 22330 hybrid_clock.cc:648] HybridClock initialized: now 1790434975740111 us; error 36 us; skew 500 ppm
Sep 26 15:02:55 dist-test-slave-jnhs krb5kdc[22317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434975, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:55.748941 22330 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:02:55.749549 22330 webserver.cc:460] Webserver started at http://127.8.102.62:39379/ using document root <none> and password file <none>
I20260926 15:02:55.749830 22330 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:55.749908 22330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:02:55.750072 22330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:02:55.751344 22330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "a008a79af4574e19aee1b8219d5f1ed3"
format_stamp: "Formatted at 2026-09-26 15:02:55 on dist-test-slave-jnhs"
server_key: "7041b229940adbcd4ead3f6d8550aaf3"
server_key_iv: "4b19a959167e0c256b1fa2323600e243"
server_key_version: "encryptionkey@0"
I20260926 15:02:55.751748 22330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "a008a79af4574e19aee1b8219d5f1ed3"
format_stamp: "Formatted at 2026-09-26 15:02:55 on dist-test-slave-jnhs"
server_key: "7041b229940adbcd4ead3f6d8550aaf3"
server_key_iv: "4b19a959167e0c256b1fa2323600e243"
server_key_version: "encryptionkey@0"
I20260926 15:02:55.753270 22330 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:02:55.754149 22345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:55.754442 22330 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:55.754535 22330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "a008a79af4574e19aee1b8219d5f1ed3"
format_stamp: "Formatted at 2026-09-26 15:02:55 on dist-test-slave-jnhs"
server_key: "7041b229940adbcd4ead3f6d8550aaf3"
server_key_iv: "4b19a959167e0c256b1fa2323600e243"
server_key_version: "encryptionkey@0"
I20260926 15:02:55.754660 22330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:55.772181 22330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:55.774951 22330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:55.775075 22330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:55.779150 22330 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:45439
I20260926 15:02:55.779210 22397 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:45439 every 8 connection(s)
I20260926 15:02:55.779582 22330 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:02:55.780599 22398 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:02:55.783011 22398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3: Bootstrap starting.
I20260926 15:02:55.783579 22398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3: Neither blocks nor log segments found. Creating new log.
I20260926 15:02:55.783830 22398 log.cc:824] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3: Log is configured to *not* fsync() on all Append() calls
I20260926 15:02:55.784557 22398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3: No bootstrap required, opened a new log
I20260926 15:02:55.786146 22398 raft_consensus.cc:348] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a008a79af4574e19aee1b8219d5f1ed3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45439 } }
I20260926 15:02:55.786293 22398 raft_consensus.cc:374] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:02:55.786317 22398 raft_consensus.cc:729] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a008a79af4574e19aee1b8219d5f1ed3, State: Initialized, Role: FOLLOWER
I20260926 15:02:55.786461 22398 consensus_queue.cc:260] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a008a79af4574e19aee1b8219d5f1ed3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45439 } }
I20260926 15:02:55.786543 22398 raft_consensus.cc:388] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:02:55.786566 22398 raft_consensus.cc:482] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:02:55.786597 22398 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:02:55.787478 22398 raft_consensus.cc:504] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a008a79af4574e19aee1b8219d5f1ed3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45439 } }
I20260926 15:02:55.787566 22398 leader_election.cc:302] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a008a79af4574e19aee1b8219d5f1ed3; no voters:
I20260926 15:02:55.787695 22398 leader_election.cc:288] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:02:55.787775 22401 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:02:55.787961 22401 raft_consensus.cc:686] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 1 LEADER]: Becoming Leader. State: Replica: a008a79af4574e19aee1b8219d5f1ed3, State: Running, Role: LEADER
I20260926 15:02:55.787987 22398 sys_catalog.cc:564] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:02:55.788127 22401 consensus_queue.cc:237] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a008a79af4574e19aee1b8219d5f1ed3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45439 } }
I20260926 15:02:55.788525 22403 sys_catalog.cc:455] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a008a79af4574e19aee1b8219d5f1ed3. Latest consensus state: current_term: 1 leader_uuid: "a008a79af4574e19aee1b8219d5f1ed3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a008a79af4574e19aee1b8219d5f1ed3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45439 } } }
I20260926 15:02:55.788512 22402 sys_catalog.cc:455] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a008a79af4574e19aee1b8219d5f1ed3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a008a79af4574e19aee1b8219d5f1ed3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45439 } } }
I20260926 15:02:55.788637 22403 sys_catalog.cc:458] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [sys.catalog]: This master's current role is: LEADER
I20260926 15:02:55.788645 22402 sys_catalog.cc:458] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [sys.catalog]: This master's current role is: LEADER
I20260926 15:02:55.788877 22410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:02:55.789119 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 22330
I20260926 15:02:55.789233 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:02:55.789459 8600 external_mini_cluster.cc:1428] Setting key 5a6b9803be20f1e764871547af7a80d9
I20260926 15:02:55.789928 22410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:02:55.791875 22410 catalog_manager.cc:1347] Generated new cluster ID: c7b7a7f3fbed451da33f3726329d8cda
I20260926 15:02:55.791941 22410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:02:55.799010 22410 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:02:55.799801 22410 catalog_manager.cc:1504] Loading token signing keys...
Sep 26 15:02:55 dist-test-slave-jnhs krb5kdc[22317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434975, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:02:55.805210 22410 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3: Generated new TSK 0
I20260926 15:02:55.805402 22410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:02:55.805820 22420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:55.790094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45216 (local address 127.8.102.62:45439)
0926 15:02:55.795335 (+ 5241us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:55.795344 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:55.795378 (+ 34us) server_negotiation.cc:407] Connection header received
0926 15:02:55.795490 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:55.795495 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:55.795541 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:55.795732 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:55.796439 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:55.797389 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:55.798168 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:55.798716 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:55.798877 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:55.801742 (+ 2865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:55.801756 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:55.801756 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:55.801769 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:55.803847 (+ 2078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:55.804267 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:55.804270 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:55.804271 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:55.804374 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:55.804567 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:55.804567 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:55.804567 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:55.804861 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:55.804979 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:55.805416 (+ 437us) server_negotiation.cc:299] Negotiation successful
0926 15:02:55.805574 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5176,"thread_start_us":70,"threads_started":1}
WARNING: no policy specified for kudu/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
I20260926 15:02:55.838312 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:45439
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:02:55.923404 22424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:55.923616 22424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:55.923642 22424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:55.924757 22424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:55.924793 22424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:55.924819 22424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:02:55.926223 22424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:45439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22424
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:55.926414 22424 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:55.926604 22424 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:55.931905 22432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:55.931905 22429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:55.931907 22430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:55.932350 22424 server_base.cc:1034] running on GCE node
I20260926 15:02:55.932601 22424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:55.932873 22424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:55.934059 22424 hybrid_clock.cc:648] HybridClock initialized: now 1790434975934043 us; error 31 us; skew 500 ppm
Sep 26 15:02:55 dist-test-slave-jnhs krb5kdc[22317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434975, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:55.937062 22424 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:02:55.937636 22424 webserver.cc:460] Webserver started at http://127.8.102.1:44749/ using document root <none> and password file <none>
I20260926 15:02:55.937918 22424 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:55.937997 22424 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:02:55.938165 22424 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:02:55.939440 22424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "7a452967a822457f83ef29ec496c8f60"
format_stamp: "Formatted at 2026-09-26 15:02:55 on dist-test-slave-jnhs"
server_key: "9fb97960cce0797b74fe48aefe3da002"
server_key_iv: "471b263716f8522cf0dc561a1449526f"
server_key_version: "encryptionkey@0"
I20260926 15:02:55.939884 22424 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "7a452967a822457f83ef29ec496c8f60"
format_stamp: "Formatted at 2026-09-26 15:02:55 on dist-test-slave-jnhs"
server_key: "9fb97960cce0797b74fe48aefe3da002"
server_key_iv: "471b263716f8522cf0dc561a1449526f"
server_key_version: "encryptionkey@0"
I20260926 15:02:55.941449 22424 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:02:55.942303 22439 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:55.942503 22424 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:55.942675 22424 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "7a452967a822457f83ef29ec496c8f60"
format_stamp: "Formatted at 2026-09-26 15:02:55 on dist-test-slave-jnhs"
server_key: "9fb97960cce0797b74fe48aefe3da002"
server_key_iv: "471b263716f8522cf0dc561a1449526f"
server_key_version: "encryptionkey@0"
I20260926 15:02:55.942776 22424 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:55.956645 22424 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:55.959124 22424 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:55.959295 22424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:55.959539 22424 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:02:55.959859 22424 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:02:55.959914 22424 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:55.959954 22424 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:02:55.959982 22424 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:55.966008 22424 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:34853
I20260926 15:02:55.966066 22552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:34853 every 8 connection(s)
I20260926 15:02:55.966363 22424 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
Sep 26 15:02:55 dist-test-slave-jnhs krb5kdc[22317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434975, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:02:55.973542 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 22424
I20260926 15:02:55.973658 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:02:55.973897 8600 external_mini_cluster.cc:1428] Setting key b593534ae6ca53515ed46284d4178a28
I20260926 15:02:55.976570 22420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:55.967155 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:49377 (local address 127.8.102.62:45439)
0926 15:02:55.967247 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:55.967249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:55.967418 (+ 169us) server_negotiation.cc:407] Connection header received
0926 15:02:55.967904 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:55.967906 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:55.967923 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:55.967990 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:55.968912 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:55.969468 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:55.970453 (+ 985us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:55.970790 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:55.970856 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:55.973036 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:55.973038 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:55.973038 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:55.973052 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:55.974851 (+ 1799us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:55.975437 (+ 586us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:55.975438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:55.975438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:55.975495 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:55.975810 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:55.975810 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:55.975810 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:55.975913 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:55.975965 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:55.976322 (+ 357us) server_negotiation.cc:299] Negotiation successful
0926 15:02:55.976439 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 15:02:55.976769 22555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:55.967039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.1:49377)
0926 15:02:55.967247 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:55.967293 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:55.967780 (+ 487us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:55.968042 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:55.968046 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:55.968619 (+ 573us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:55.968818 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:55.968823 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:55.969551 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:55.969552 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:55.970360 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:55.970362 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:55.970874 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:55.970875 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:55.970988 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:55.971502 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:55.971511 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:55.972933 (+ 1422us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:55.974946 (+ 2013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:55.974954 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:55.974956 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:55.975348 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:55.975571 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:55.975572 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:55.975572 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:55.975728 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:55.975978 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:55.975984 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:55.976180 (+ 196us) client_negotiation.cc:769] Sending connection context
0926 15:02:55.976299 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 15:02:55.976492 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":98,"threads_started":1}
I20260926 15:02:55.977314 22553 heartbeater.cc:347] Connected to a master server at 127.8.102.62:45439
I20260926 15:02:55.977438 22553 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:55.977641 22553 heartbeater.cc:511] Master 127.8.102.62:45439 requested a full tablet report, sending...
I20260926 15:02:55.978194 22362 ts_manager.cc:194] Registered new tserver with Master: 7a452967a822457f83ef29ec496c8f60 (127.8.102.1:34853)
I20260926 15:02:55.979179 22362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:49377
WARNING: no policy specified for kudu/127.8.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
I20260926 15:02:56.002492 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:45439
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:02:56.085786 22558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:56.086076 22558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:56.086122 22558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:56.087467 22558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:56.087554 22558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:56.087610 22558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:02:56.089504 22558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:45439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22558
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:56.089814 22558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:56.090080 22558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:56.096071 22566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:56.096071 22564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:56.096161 22558 server_base.cc:1034] running on GCE node
W20260926 15:02:56.096091 22563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:56.096544 22558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:56.096858 22558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:56.098040 22558 hybrid_clock.cc:648] HybridClock initialized: now 1790434976098017 us; error 33 us; skew 500 ppm
Sep 26 15:02:56 dist-test-slave-jnhs krb5kdc[22317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434976, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:56.100920 22558 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:02:56.101488 22558 webserver.cc:460] Webserver started at http://127.8.102.2:33135/ using document root <none> and password file <none>
I20260926 15:02:56.101747 22558 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:56.101819 22558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:02:56.101984 22558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:02:56.103220 22558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "43ca9df1eb32403fad905322acf736ca"
format_stamp: "Formatted at 2026-09-26 15:02:56 on dist-test-slave-jnhs"
server_key: "38fca355a3c10d9d276fee24d98a9c9a"
server_key_iv: "5a4fbddc1a9d1cedbe6e0e7aa1329c07"
server_key_version: "encryptionkey@0"
I20260926 15:02:56.103622 22558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "43ca9df1eb32403fad905322acf736ca"
format_stamp: "Formatted at 2026-09-26 15:02:56 on dist-test-slave-jnhs"
server_key: "38fca355a3c10d9d276fee24d98a9c9a"
server_key_iv: "5a4fbddc1a9d1cedbe6e0e7aa1329c07"
server_key_version: "encryptionkey@0"
I20260926 15:02:56.105038 22558 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:02:56.105800 22573 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:56.105973 22558 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 15:02:56.106078 22558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "43ca9df1eb32403fad905322acf736ca"
format_stamp: "Formatted at 2026-09-26 15:02:56 on dist-test-slave-jnhs"
server_key: "38fca355a3c10d9d276fee24d98a9c9a"
server_key_iv: "5a4fbddc1a9d1cedbe6e0e7aa1329c07"
server_key_version: "encryptionkey@0"
I20260926 15:02:56.106206 22558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:56.132162 22558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:56.134886 22558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:56.135066 22558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:56.135382 22558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:02:56.135744 22558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:02:56.135808 22558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:56.135854 22558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:02:56.135888 22558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:56.142308 22558 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:36209
I20260926 15:02:56.142349 22686 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:36209 every 8 connection(s)
I20260926 15:02:56.142738 22558 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:02:56.148094 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 22558
I20260926 15:02:56.148218 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:02:56.148439 8600 external_mini_cluster.cc:1428] Setting key 12d6897f89eb27b70d45c40ef3a0b6b0
Sep 26 15:02:56 dist-test-slave-jnhs krb5kdc[22317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434976, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:02:56.154001 22420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.143698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:55229 (local address 127.8.102.62:45439)
0926 15:02:56.143794 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:56.143797 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:56.143999 (+ 202us) server_negotiation.cc:407] Connection header received
0926 15:02:56.144655 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:56.144657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:56.144670 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:56.144729 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:56.145913 (+ 1184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.146463 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.147682 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.147981 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.148069 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:56.150480 (+ 2411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:56.150482 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:56.150483 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:56.150490 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:56.152348 (+ 1858us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:56.152821 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:56.152822 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:56.152822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:56.152889 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:56.153219 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:56.153220 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:56.153221 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:56.153403 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:56.153478 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:56.153784 (+ 306us) server_negotiation.cc:299] Negotiation successful
0926 15:02:56.153884 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260926 15:02:56.154096 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.143584 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.2:55229)
0926 15:02:56.143843 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:56.143901 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:56.144524 (+ 623us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:56.144810 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:56.144814 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:56.145587 (+ 773us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:56.145804 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:56.145810 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.146673 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.146674 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:56.147579 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:56.147581 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.148094 (+ 513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.148095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:56.148257 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:56.148897 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:56.148908 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:56.150375 (+ 1467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:56.152444 (+ 2069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:56.152451 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:56.152453 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:56.152736 (+ 283us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:56.152963 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:56.152963 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:56.152964 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:56.153127 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:56.153477 (+ 350us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:56.153483 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:56.153617 (+ 134us) client_negotiation.cc:769] Sending connection context
0926 15:02:56.153736 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 15:02:56.153876 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"thread_start_us":111,"threads_started":1}
I20260926 15:02:56.154681 22687 heartbeater.cc:347] Connected to a master server at 127.8.102.62:45439
I20260926 15:02:56.154789 22687 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:56.155012 22687 heartbeater.cc:511] Master 127.8.102.62:45439 requested a full tablet report, sending...
I20260926 15:02:56.155452 22362 ts_manager.cc:194] Registered new tserver with Master: 43ca9df1eb32403fad905322acf736ca (127.8.102.2:36209)
I20260926 15:02:56.155834 22362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:55229
WARNING: no policy specified for kudu/127.8.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
I20260926 15:02:56.177402 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:45439
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:02:56.259266 22692 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:56.259509 22692 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:56.259550 22692 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:56.260756 22692 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:56.260828 22692 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:56.260876 22692 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:02:56.262377 22692 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:45439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22692
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:56.262710 22692 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:56.262948 22692 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:56.268151 22700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:56.268208 22698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:56.268170 22697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:56.268383 22692 server_base.cc:1034] running on GCE node
I20260926 15:02:56.268631 22692 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:56.268877 22692 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:56.270037 22692 hybrid_clock.cc:648] HybridClock initialized: now 1790434976270015 us; error 42 us; skew 500 ppm
Sep 26 15:02:56 dist-test-slave-jnhs krb5kdc[22317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434976, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:56.272713 22692 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:02:56.273213 22692 webserver.cc:460] Webserver started at http://127.8.102.3:36329/ using document root <none> and password file <none>
I20260926 15:02:56.273433 22692 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:56.273474 22692 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:02:56.273571 22692 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:02:56.274606 22692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "0eabcd36a11845d5b9bc40918437fa3b"
format_stamp: "Formatted at 2026-09-26 15:02:56 on dist-test-slave-jnhs"
server_key: "223b5a3b280223492d3b75e87232a83d"
server_key_iv: "5215461b21f1207046b2c40b5770ef7d"
server_key_version: "encryptionkey@0"
I20260926 15:02:56.275055 22692 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "0eabcd36a11845d5b9bc40918437fa3b"
format_stamp: "Formatted at 2026-09-26 15:02:56 on dist-test-slave-jnhs"
server_key: "223b5a3b280223492d3b75e87232a83d"
server_key_iv: "5215461b21f1207046b2c40b5770ef7d"
server_key_version: "encryptionkey@0"
I20260926 15:02:56.276407 22692 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:56.277218 22707 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:56.277418 22692 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:56.277499 22692 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "0eabcd36a11845d5b9bc40918437fa3b"
format_stamp: "Formatted at 2026-09-26 15:02:56 on dist-test-slave-jnhs"
server_key: "223b5a3b280223492d3b75e87232a83d"
server_key_iv: "5215461b21f1207046b2c40b5770ef7d"
server_key_version: "encryptionkey@0"
I20260926 15:02:56.277549 22692 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:56.300240 22692 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:56.302452 22692 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:56.302579 22692 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:56.302891 22692 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:02:56.303194 22692 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:02:56.303226 22692 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:56.303249 22692 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:02:56.303262 22692 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:56.309231 22692 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:41671
I20260926 15:02:56.309438 22820 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:41671 every 8 connection(s)
I20260926 15:02:56.309707 22692 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:02:56.312813 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 22692
I20260926 15:02:56.312917 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:02:56.313114 8600 external_mini_cluster.cc:1428] Setting key 081170110228096307115fc258188217
Sep 26 15:02:56 dist-test-slave-jnhs krb5kdc[22317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434976, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:02:56.320189 22420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.310587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:49379 (local address 127.8.102.62:45439)
0926 15:02:56.310689 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:56.310692 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:56.310930 (+ 238us) server_negotiation.cc:407] Connection header received
0926 15:02:56.311440 (+ 510us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:56.311442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:56.311454 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:56.311520 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:56.312660 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.313503 (+ 843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.314782 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.314975 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.315039 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:56.317321 (+ 2282us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:56.317322 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:56.317323 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:56.317330 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:56.318753 (+ 1423us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:56.319162 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:56.319163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:56.319163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:56.319220 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:56.319485 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:56.319487 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:56.319487 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:56.319614 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:56.319716 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:56.320009 (+ 293us) server_negotiation.cc:299] Negotiation successful
0926 15:02:56.320093 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260926 15:02:56.320559 22823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.310550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.3:49379)
0926 15:02:56.310796 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:56.310837 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:56.311323 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:56.311632 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:56.311636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:56.312295 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:56.312508 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:56.312513 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.313635 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.313635 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:56.314683 (+ 1048us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:56.314687 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.315093 (+ 406us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.315094 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:56.315202 (+ 108us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:56.315706 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:56.315716 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:56.317220 (+ 1504us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:56.318846 (+ 1626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:56.318853 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:56.318855 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:56.319084 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:56.319305 (+ 221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:56.319306 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:56.319306 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:56.319418 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:56.319709 (+ 291us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:56.319716 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:56.319870 (+ 154us) client_negotiation.cc:769] Sending connection context
0926 15:02:56.319975 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 15:02:56.320153 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"thread_start_us":117,"threads_started":1}
I20260926 15:02:56.321110 22821 heartbeater.cc:347] Connected to a master server at 127.8.102.62:45439
I20260926 15:02:56.321195 22821 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:56.321341 22821 heartbeater.cc:511] Master 127.8.102.62:45439 requested a full tablet report, sending...
I20260926 15:02:56.321683 22362 ts_manager.cc:194] Registered new tserver with Master: 0eabcd36a11845d5b9bc40918437fa3b (127.8.102.3:41671)
I20260926 15:02:56.322023 22362 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:49379
I20260926 15:02:56.324950 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:02:56.333200 22420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.326370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45228 (local address 127.8.102.62:45439)
0926 15:02:56.326430 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:56.326432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:56.326585 (+ 153us) server_negotiation.cc:407] Connection header received
0926 15:02:56.326691 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:56.326692 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:56.326702 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:56.326760 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:56.327342 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.327814 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.328573 (+ 759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.328766 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.328822 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:56.330303 (+ 1481us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:56.330304 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:56.330305 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:56.330311 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:56.331950 (+ 1639us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:56.332316 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:56.332317 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:56.332317 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:56.332369 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:56.332592 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:56.332594 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:56.332594 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:56.332771 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:56.332869 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:56.332973 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 15:02:56.333046 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 15:02:56.335044 22362 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45228:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:02:56.336875 22362 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:02:56.343652 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.339629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54046 (local address 127.8.102.3:41671)
0926 15:02:56.339818 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:56.339824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:56.339836 (+ 12us) server_negotiation.cc:407] Connection header received
0926 15:02:56.339881 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:56.339883 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:56.339900 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:56.339977 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:02:56.340304 (+ 327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.341138 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.342504 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.343232 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.343325 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:56.343388 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:56.343521 (+ 133us) server_negotiation.cc:299] Negotiation successful
0926 15:02:56.343559 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":118,"threads_started":1}
I20260926 15:02:56.343639 22832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.339421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:41671 (local address 127.0.0.1:54046)
0926 15:02:56.339608 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:56.339652 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:56.339720 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:56.340091 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:56.340095 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:56.340097 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:02:56.340172 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:56.340174 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.341262 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.341263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:56.342411 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:56.342414 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.343327 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.343327 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:56.343412 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:56.343426 (+ 14us) client_negotiation.cc:769] Sending connection context
0926 15:02:56.343499 (+ 73us) client_negotiation.cc:240] Negotiation successful
0926 15:02:56.343558 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":66,"threads_started":1}
I20260926 15:02:56.344107 22833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.339474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:36209 (local address 127.0.0.1:59070)
0926 15:02:56.339836 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:56.339848 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:56.339899 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:56.340448 (+ 549us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:56.340449 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:56.340451 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:02:56.340482 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:56.340484 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.341772 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.341773 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:56.342848 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:56.342850 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.343885 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.343887 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:56.343943 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:56.343946 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:02:56.343990 (+ 44us) client_negotiation.cc:240] Negotiation successful
0926 15:02:56.344032 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":22,"thread_start_us":87,"threads_started":1}
I20260926 15:02:56.344424 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.339601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59070 (local address 127.8.102.2:36209)
0926 15:02:56.340227 (+ 626us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:56.340232 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:56.340242 (+ 10us) server_negotiation.cc:407] Connection header received
0926 15:02:56.340277 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:56.340278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:56.340295 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:56.340366 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:02:56.340557 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.341649 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.342957 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.343701 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.343784 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:56.343852 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:56.344290 (+ 438us) server_negotiation.cc:299] Negotiation successful
0926 15:02:56.344343 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":73,"threads_started":1}
I20260926 15:02:56.344399 22755 tablet_service.cc:1467] Processing CreateTablet for tablet 35ec2dfb00a94e7aa6433fc419670c09 (DEFAULT_TABLE table=test-table [id=3f9853eb26c741abb71ff1f9619653f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:02:56.344808 22755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35ec2dfb00a94e7aa6433fc419670c09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:02:56.344920 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.339625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:34853 (local address 127.0.0.1:43054)
0926 15:02:56.340035 (+ 410us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:56.340045 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:56.340101 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:56.340369 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:56.340370 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:56.340372 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:02:56.340417 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:56.340418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.341833 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.341834 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:56.342978 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:56.342980 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.344677 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.344677 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:56.344745 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:56.344750 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:02:56.344812 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 15:02:56.344856 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":90,"threads_started":1}
I20260926 15:02:56.345000 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.339852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43054 (local address 127.8.102.1:34853)
0926 15:02:56.340027 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:56.340032 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:56.340095 (+ 63us) server_negotiation.cc:407] Connection header received
0926 15:02:56.340248 (+ 153us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:56.340251 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:56.340266 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:56.340489 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:02:56.340559 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.341689 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.343064 (+ 1375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.344101 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.344359 (+ 258us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:56.344425 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:56.344841 (+ 416us) server_negotiation.cc:299] Negotiation successful
0926 15:02:56.344921 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":95,"threads_started":1}
I20260926 15:02:56.345131 22621 tablet_service.cc:1467] Processing CreateTablet for tablet 35ec2dfb00a94e7aa6433fc419670c09 (DEFAULT_TABLE table=test-table [id=3f9853eb26c741abb71ff1f9619653f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:02:56.345394 22621 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35ec2dfb00a94e7aa6433fc419670c09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:02:56.347995 22838 tablet_bootstrap.cc:492] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b: Bootstrap starting.
I20260926 15:02:56.348263 22839 tablet_bootstrap.cc:492] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca: Bootstrap starting.
I20260926 15:02:56.348348 22487 tablet_service.cc:1467] Processing CreateTablet for tablet 35ec2dfb00a94e7aa6433fc419670c09 (DEFAULT_TABLE table=test-table [id=3f9853eb26c741abb71ff1f9619653f5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:02:56.348659 22487 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 35ec2dfb00a94e7aa6433fc419670c09. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:02:56.348680 22838 tablet_bootstrap.cc:654] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b: Neither blocks nor log segments found. Creating new log.
I20260926 15:02:56.348860 22839 tablet_bootstrap.cc:654] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca: Neither blocks nor log segments found. Creating new log.
I20260926 15:02:56.349315 22838 log.cc:824] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b: Log is configured to *not* fsync() on all Append() calls
I20260926 15:02:56.349426 22839 log.cc:824] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca: Log is configured to *not* fsync() on all Append() calls
I20260926 15:02:56.350067 22838 tablet_bootstrap.cc:492] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b: No bootstrap required, opened a new log
I20260926 15:02:56.350122 22838 ts_tablet_manager.cc:1397] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:02:56.350308 22839 tablet_bootstrap.cc:492] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca: No bootstrap required, opened a new log
I20260926 15:02:56.350440 22839 ts_tablet_manager.cc:1397] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:02:56.351722 22838 raft_consensus.cc:348] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } }
I20260926 15:02:56.351917 22838 raft_consensus.cc:374] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:02:56.351967 22838 raft_consensus.cc:729] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eabcd36a11845d5b9bc40918437fa3b, State: Initialized, Role: FOLLOWER
I20260926 15:02:56.351958 22842 tablet_bootstrap.cc:492] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60: Bootstrap starting.
I20260926 15:02:56.352097 22838 consensus_queue.cc:260] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } }
I20260926 15:02:56.352392 22838 ts_tablet_manager.cc:1428] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260926 15:02:56.352512 22821 heartbeater.cc:503] Master 127.8.102.62:45439 was elected leader, sending a full tablet report...
I20260926 15:02:56.352667 22839 raft_consensus.cc:348] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } }
I20260926 15:02:56.352864 22842 tablet_bootstrap.cc:654] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60: Neither blocks nor log segments found. Creating new log.
I20260926 15:02:56.352869 22839 raft_consensus.cc:374] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:02:56.352916 22839 raft_consensus.cc:729] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ca9df1eb32403fad905322acf736ca, State: Initialized, Role: FOLLOWER
I20260926 15:02:56.353027 22839 consensus_queue.cc:260] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } }
I20260926 15:02:56.353302 22839 ts_tablet_manager.cc:1428] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:02:56.353339 22687 heartbeater.cc:503] Master 127.8.102.62:45439 was elected leader, sending a full tablet report...
I20260926 15:02:56.353431 22842 log.cc:824] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60: Log is configured to *not* fsync() on all Append() calls
I20260926 15:02:56.354256 22842 tablet_bootstrap.cc:492] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60: No bootstrap required, opened a new log
I20260926 15:02:56.354352 22842 ts_tablet_manager.cc:1397] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:02:56.355885 22842 raft_consensus.cc:348] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } }
I20260926 15:02:56.356043 22842 raft_consensus.cc:374] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:02:56.356066 22842 raft_consensus.cc:729] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a452967a822457f83ef29ec496c8f60, State: Initialized, Role: FOLLOWER
I20260926 15:02:56.356213 22842 consensus_queue.cc:260] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } }
I20260926 15:02:56.356496 22842 ts_tablet_manager.cc:1428] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:02:56.356499 22553 heartbeater.cc:503] Master 127.8.102.62:45439 was elected leader, sending a full tablet report...
W20260926 15:02:56.393421 22688 tablet.cc:2369] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:02:56.467167 22554 tablet.cc:2369] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:02:56.511787 22844 raft_consensus.cc:482] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:02:56.511929 22844 raft_consensus.cc:504] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } }
I20260926 15:02:56.512283 22844 leader_election.cc:288] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0eabcd36a11845d5b9bc40918437fa3b (127.8.102.3:41671), 7a452967a822457f83ef29ec496c8f60 (127.8.102.1:34853)
I20260926 15:02:56.516820 22689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.512636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:41671 (local address 127.8.102.2:46719)
0926 15:02:56.512866 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:56.512881 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:56.512962 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:56.513192 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:56.513193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:56.513195 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:02:56.513273 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:56.513275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.514206 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.514207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:56.515629 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:56.515633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.516515 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.516516 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:56.516606 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:56.516609 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:02:56.516662 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 15:02:56.516718 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":108}
I20260926 15:02:56.516847 22836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.512733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:46719 (local address 127.8.102.3:41671)
0926 15:02:56.512898 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:56.512900 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:56.512966 (+ 66us) server_negotiation.cc:407] Connection header received
0926 15:02:56.513055 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:56.513056 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:56.513071 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:56.513156 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:02:56.513353 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.514101 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.515755 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.516428 (+ 673us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.516537 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:56.516553 (+ 16us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:56.516716 (+ 163us) server_negotiation.cc:299] Negotiation successful
0926 15:02:56.516770 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140}
I20260926 15:02:56.516855 22837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.512877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:38879 (local address 127.8.102.1:34853)
0926 15:02:56.512924 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:56.512927 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:56.513060 (+ 133us) server_negotiation.cc:407] Connection header received
0926 15:02:56.513139 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:56.513140 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:56.513153 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:56.513205 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:02:56.513349 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.514105 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.515755 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.516352 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.516444 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:56.516457 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:56.516716 (+ 259us) server_negotiation.cc:299] Negotiation successful
0926 15:02:56.516770 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 15:02:56.516827 22847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.512811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:34853 (local address 127.8.102.2:38879)
0926 15:02:56.512982 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:56.512992 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:56.513079 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:56.513204 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:56.513204 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:56.513205 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:02:56.513273 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:56.513275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.514206 (+ 931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.514207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:56.515629 (+ 1422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:56.515633 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.516494 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.516496 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:56.516585 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:56.516591 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 15:02:56.516659 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 15:02:56.516715 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":149,"mutex_wait_us":82,"thread_start_us":113,"threads_started":1}
I20260926 15:02:56.517320 22507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ec2dfb00a94e7aa6433fc419670c09" candidate_uuid: "43ca9df1eb32403fad905322acf736ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a452967a822457f83ef29ec496c8f60" is_pre_election: true
I20260926 15:02:56.517544 22507 raft_consensus.cc:2454] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43ca9df1eb32403fad905322acf736ca in term 0.
I20260926 15:02:56.517482 22775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ec2dfb00a94e7aa6433fc419670c09" candidate_uuid: "43ca9df1eb32403fad905322acf736ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eabcd36a11845d5b9bc40918437fa3b" is_pre_election: true
I20260926 15:02:56.517696 22775 raft_consensus.cc:2454] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43ca9df1eb32403fad905322acf736ca in term 0.
I20260926 15:02:56.517867 22576 leader_election.cc:302] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43ca9df1eb32403fad905322acf736ca, 7a452967a822457f83ef29ec496c8f60; no voters:
I20260926 15:02:56.518056 22844 raft_consensus.cc:2781] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:02:56.518148 22844 raft_consensus.cc:482] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:02:56.518183 22844 raft_consensus.cc:3037] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:02:56.519039 22844 raft_consensus.cc:504] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } }
I20260926 15:02:56.519215 22844 leader_election.cc:288] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [CANDIDATE]: Term 1 election: Requested vote from peers 0eabcd36a11845d5b9bc40918437fa3b (127.8.102.3:41671), 7a452967a822457f83ef29ec496c8f60 (127.8.102.1:34853)
I20260926 15:02:56.519428 22775 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ec2dfb00a94e7aa6433fc419670c09" candidate_uuid: "43ca9df1eb32403fad905322acf736ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eabcd36a11845d5b9bc40918437fa3b"
I20260926 15:02:56.519486 22507 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "35ec2dfb00a94e7aa6433fc419670c09" candidate_uuid: "43ca9df1eb32403fad905322acf736ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a452967a822457f83ef29ec496c8f60"
I20260926 15:02:56.519536 22775 raft_consensus.cc:3037] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:02:56.519585 22507 raft_consensus.cc:3037] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:02:56.520689 22775 raft_consensus.cc:2454] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43ca9df1eb32403fad905322acf736ca in term 1.
I20260926 15:02:56.520720 22507 raft_consensus.cc:2454] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43ca9df1eb32403fad905322acf736ca in term 1.
I20260926 15:02:56.520903 22576 leader_election.cc:302] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43ca9df1eb32403fad905322acf736ca, 7a452967a822457f83ef29ec496c8f60; no voters:
I20260926 15:02:56.521023 22844 raft_consensus.cc:2781] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:02:56.521210 22844 raft_consensus.cc:686] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [term 1 LEADER]: Becoming Leader. State: Replica: 43ca9df1eb32403fad905322acf736ca, State: Running, Role: LEADER
I20260926 15:02:56.521330 22844 consensus_queue.cc:237] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } }
I20260926 15:02:56.522285 22361 catalog_manager.cc:5495] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 43ca9df1eb32403fad905322acf736ca (127.8.102.2). New cstate: current_term: 1 leader_uuid: "43ca9df1eb32403fad905322acf736ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } health_report { overall_health: HEALTHY } } }
W20260926 15:02:56.560387 22822 tablet.cc:2369] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:02:56.571473 22834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.568791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59084 (local address 127.8.102.2:36209)
0926 15:02:56.568870 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:56.568873 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:56.568894 (+ 21us) server_negotiation.cc:407] Connection header received
0926 15:02:56.568936 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:56.568937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:56.568959 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:56.569125 (+ 166us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:02:56.569287 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.569813 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.570609 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.570839 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.570919 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:56.571093 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:02:56.571206 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:02:56.571303 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:56.571342 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 15:02:56.571363 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260926 15:02:56.573263 22775 raft_consensus.cc:1261] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b [term 1 FOLLOWER]: Refusing update from remote peer 43ca9df1eb32403fad905322acf736ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:02:56.573288 22507 raft_consensus.cc:1261] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Refusing update from remote peer 43ca9df1eb32403fad905322acf736ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:02:56.573663 22848 consensus_queue.cc:1035] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:02:56.573808 22844 consensus_queue.cc:1035] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:02:56.579562 22362 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:02:56.579702 22362 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:45228:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:02:56.580566 22362 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3: Sending DeleteTablet for 3 replicas of tablet 35ec2dfb00a94e7aa6433fc419670c09
I20260926 15:02:56.580955 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 22424
I20260926 15:02:56.581049 22487 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ec2dfb00a94e7aa6433fc419670c09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:02:56 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:43054
I20260926 15:02:56.581080 22755 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ec2dfb00a94e7aa6433fc419670c09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:02:56 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:54046
I20260926 15:02:56.582087 22621 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ec2dfb00a94e7aa6433fc419670c09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:02:56 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:59070
I20260926 15:02:56.582124 22861 tablet_replica.cc:330] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b: stopping tablet replica
I20260926 15:02:56.582235 22861 raft_consensus.cc:2229] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:02:56.582346 22861 raft_consensus.cc:2258] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:02:56.582939 22861 ts_tablet_manager.cc:1905] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:02:56.584860 22861 ts_tablet_manager.cc:1918] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:02:56.584969 22861 log.cc:1196] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/35ec2dfb00a94e7aa6433fc419670c09
I20260926 15:02:56.585251 22861 ts_tablet_manager.cc:1939] T 35ec2dfb00a94e7aa6433fc419670c09 P 0eabcd36a11845d5b9bc40918437fa3b: Deleting consensus metadata
I20260926 15:02:56.585702 22348 catalog_manager.cc:4841] TS 0eabcd36a11845d5b9bc40918437fa3b (127.8.102.3:41671): tablet 35ec2dfb00a94e7aa6433fc419670c09 (table test-table [id=3f9853eb26c741abb71ff1f9619653f5]) successfully deleted
W20260926 15:02:56.586665 22348 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.102.1:34853 (error 108)
W20260926 15:02:56.587576 22835 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:02:56.587094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:34853 (local address 127.0.0.1:43066)
0926 15:02:56.587151 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:56.587163 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:56.587204 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:56.587540 (+ 336us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.1:34853: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 15:02:56.587802 22348 catalog_manager.cc:4584] TS 7a452967a822457f83ef29ec496c8f60 (127.8.102.1:34853): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 35ec2dfb00a94e7aa6433fc419670c09: Network error: Client connection negotiation failed: client connection to 127.8.102.1:34853: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 15:02:56.587852 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 22558
I20260926 15:02:56.593972 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 22692
I20260926 15:02:56.594146 22835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.593971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:36209 (local address 127.0.0.1:59096)
0926 15:02:56.594048 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:56.594101 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.2:36209: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260926 15:02:56.598974 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 22330
I20260926 15:02:56.604434 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:45439
--webserver_interface=127.8.102.62
--webserver_port=39379
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:45439
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:02:56.689958 22864 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:56.690176 22864 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:56.690203 22864 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:56.691437 22864 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:02:56.691481 22864 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:56.691495 22864 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:56.691509 22864 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:02:56.691520 22864 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:02:56.692958 22864 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:45439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:45439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=39379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22864
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:56.693166 22864 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:56.693341 22864 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:56.702904 22870 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:56.702910 22873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:56.702970 22871 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:56.702944 22864 server_base.cc:1034] running on GCE node
I20260926 15:02:56.703374 22864 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:56.703737 22864 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:56.704921 22864 hybrid_clock.cc:648] HybridClock initialized: now 1790434976704900 us; error 52 us; skew 500 ppm
Sep 26 15:02:56 dist-test-slave-jnhs krb5kdc[22317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434976, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:56.707839 22864 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:02:56.708281 22864 webserver.cc:460] Webserver started at http://127.8.102.62:39379/ using document root <none> and password file <none>
I20260926 15:02:56.708510 22864 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:56.708564 22864 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:02:56.710091 22864 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 15:02:56.710918 22880 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:56.711130 22864 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:02:56.711233 22864 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "a008a79af4574e19aee1b8219d5f1ed3"
format_stamp: "Formatted at 2026-09-26 15:02:55 on dist-test-slave-jnhs"
server_key: "7041b229940adbcd4ead3f6d8550aaf3"
server_key_iv: "4b19a959167e0c256b1fa2323600e243"
server_key_version: "encryptionkey@0"
I20260926 15:02:56.711647 22864 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:56.743296 22864 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:56.745472 22864 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:56.745603 22864 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:56.749501 22864 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:45439
I20260926 15:02:56.749612 22932 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:45439 every 8 connection(s)
I20260926 15:02:56.749970 22864 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:02:56.751688 22933 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:02:56.752368 22933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3: Bootstrap starting.
I20260926 15:02:56.760093 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 22864
I20260926 15:02:56.760454 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:34853
--local_ip_for_outbound_sockets=127.8.102.1
--tserver_master_addrs=127.8.102.62:45439
--webserver_port=44749
--webserver_interface=127.8.102.1
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:02:56.760851 22933 log.cc:824] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3: Log is configured to *not* fsync() on all Append() calls
I20260926 15:02:56.763677 22933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:02:56.764016 22933 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3: Bootstrap complete.
I20260926 15:02:56.766516 22933 raft_consensus.cc:348] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a008a79af4574e19aee1b8219d5f1ed3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45439 } }
I20260926 15:02:56.766885 22933 raft_consensus.cc:729] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a008a79af4574e19aee1b8219d5f1ed3, State: Initialized, Role: FOLLOWER
I20260926 15:02:56.767084 22933 consensus_queue.cc:260] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a008a79af4574e19aee1b8219d5f1ed3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45439 } }
I20260926 15:02:56.767170 22933 raft_consensus.cc:388] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:02:56.767232 22933 raft_consensus.cc:482] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:02:56.767287 22933 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 1 FOLLOWER]: Advancing to term 2
I20260926 15:02:56.768438 22933 raft_consensus.cc:504] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a008a79af4574e19aee1b8219d5f1ed3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45439 } }
I20260926 15:02:56.768585 22933 leader_election.cc:302] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a008a79af4574e19aee1b8219d5f1ed3; no voters:
I20260926 15:02:56.768805 22933 leader_election.cc:288] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 15:02:56.768877 22937 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 2 FOLLOWER]: Leader election won for term 2
I20260926 15:02:56.769042 22937 raft_consensus.cc:686] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 2 LEADER]: Becoming Leader. State: Replica: a008a79af4574e19aee1b8219d5f1ed3, State: Running, Role: LEADER
I20260926 15:02:56.769110 22933 sys_catalog.cc:564] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:02:56.769152 22937 consensus_queue.cc:237] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a008a79af4574e19aee1b8219d5f1ed3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45439 } }
I20260926 15:02:56.769435 22937 sys_catalog.cc:455] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a008a79af4574e19aee1b8219d5f1ed3. Latest consensus state: current_term: 2 leader_uuid: "a008a79af4574e19aee1b8219d5f1ed3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a008a79af4574e19aee1b8219d5f1ed3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45439 } } }
I20260926 15:02:56.769544 22937 sys_catalog.cc:458] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [sys.catalog]: This master's current role is: LEADER
I20260926 15:02:56.769436 22938 sys_catalog.cc:455] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "a008a79af4574e19aee1b8219d5f1ed3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a008a79af4574e19aee1b8219d5f1ed3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45439 } } }
I20260926 15:02:56.769750 22938 sys_catalog.cc:458] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [sys.catalog]: This master's current role is: LEADER
I20260926 15:02:56.769769 22942 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:02:56.770964 22942 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:02:56.771090 22942 catalog_manager.cc:1259] Loaded cluster ID: c7b7a7f3fbed451da33f3726329d8cda
I20260926 15:02:56.771239 22942 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:02:56.771657 22942 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260926 15:02:56.776881 22942 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260926 15:02:56.780056 22942 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:02:56.780292 22942 catalog_manager.cc:5879] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3: Loaded TSK: 0
I20260926 15:02:56.780562 22942 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 15:02:56.848242 22936 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:56.848501 22936 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:56.848543 22936 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:56.849771 22936 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:56.849853 22936 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:56.849900 22936 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:02:56.851452 22936 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:34853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=44749
--tserver_master_addrs=127.8.102.62:45439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22936
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:56.851743 22936 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:56.851989 22936 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:56.858506 22963 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:56.858475 22961 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:56.858511 22960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:56.859321 22936 server_base.cc:1034] running on GCE node
I20260926 15:02:56.859611 22936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:56.859932 22936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:56.861138 22936 hybrid_clock.cc:648] HybridClock initialized: now 1790434976861097 us; error 55 us; skew 500 ppm
Sep 26 15:02:56 dist-test-slave-jnhs krb5kdc[22317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434976, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:56.864781 22936 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:02:56.865417 22936 webserver.cc:460] Webserver started at http://127.8.102.1:44749/ using document root <none> and password file <none>
I20260926 15:02:56.865758 22936 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:56.865860 22936 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:02:56.867975 22936 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:56.868855 22970 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:56.869124 22936 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:56.869220 22936 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "7a452967a822457f83ef29ec496c8f60"
format_stamp: "Formatted at 2026-09-26 15:02:55 on dist-test-slave-jnhs"
server_key: "9fb97960cce0797b74fe48aefe3da002"
server_key_iv: "471b263716f8522cf0dc561a1449526f"
server_key_version: "encryptionkey@0"
I20260926 15:02:56.869562 22936 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:56.881217 22936 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:56.883608 22936 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:56.883772 22936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:56.884028 22936 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:02:56.884486 22977 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 15:02:56.886613 22936 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 15:02:56.886716 22936 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 15:02:56.886775 22936 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 15:02:56.887650 22936 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 15:02:56.887693 22936 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 15:02:56.887739 22977 tablet_bootstrap.cc:492] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60: Bootstrap starting.
I20260926 15:02:56.895625 22936 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:34853
I20260926 15:02:56.896188 22936 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:02:56.896546 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 22936
I20260926 15:02:56.896915 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:36209
--local_ip_for_outbound_sockets=127.8.102.2
--tserver_master_addrs=127.8.102.62:45439
--webserver_port=33135
--webserver_interface=127.8.102.2
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:02:56.899119 22977 log.cc:824] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60: Log is configured to *not* fsync() on all Append() calls
I20260926 15:02:56.902798 22977 tablet_bootstrap.cc:492] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:02:56.903151 22977 tablet_bootstrap.cc:492] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60: Bootstrap complete.
I20260926 15:02:56.904512 22977 ts_tablet_manager.cc:1397] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.009s
I20260926 15:02:56.904803 23084 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:34853 every 8 connection(s)
I20260926 15:02:56.907027 22977 raft_consensus.cc:348] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } }
I20260926 15:02:56.907385 22977 raft_consensus.cc:729] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a452967a822457f83ef29ec496c8f60, State: Initialized, Role: FOLLOWER
I20260926 15:02:56.907544 22977 consensus_queue.cc:260] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } }
I20260926 15:02:56.907831 22977 ts_tablet_manager.cc:1428] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60: Time spent starting tablet: real 0.003s user 0.005s sys 0.000s
Sep 26 15:02:56 dist-test-slave-jnhs krb5kdc[22317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434976, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:02:56.920157 23089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.905590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.1:35997)
0926 15:02:56.906927 (+ 1337us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:56.906978 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:56.907244 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:56.908104 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:56.908108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:56.908466 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:56.908728 (+ 262us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:56.908734 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.910492 (+ 1758us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.910493 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:56.911785 (+ 1292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:56.911788 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.912692 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.912693 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:56.912827 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:56.913484 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:56.913503 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:56.915548 (+ 2045us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:56.918204 (+ 2656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:56.918211 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:56.918213 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:56.918515 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:56.918872 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:56.918873 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:56.918873 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:56.919006 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:56.919515 (+ 509us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:56.919521 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:56.919674 (+ 153us) client_negotiation.cc:769] Sending connection context
0926 15:02:56.919774 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 15:02:56.919943 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1291,"thread_start_us":101,"threads_started":1}
I20260926 15:02:56.920219 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.905682 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:35997 (local address 127.8.102.62:45439)
0926 15:02:56.906799 (+ 1117us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:56.906811 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:56.907114 (+ 303us) server_negotiation.cc:407] Connection header received
0926 15:02:56.907620 (+ 506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:56.907692 (+ 72us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:56.907753 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:56.907864 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:56.908855 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.910381 (+ 1526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.911879 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.912591 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.912712 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:56.915678 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:56.915684 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:56.915684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:56.915700 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:56.918096 (+ 2396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:56.918664 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:56.918668 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:56.918669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:56.918796 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:56.919102 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:56.919103 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:56.919103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:56.919433 (+ 330us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:56.919523 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:56.919869 (+ 346us) server_negotiation.cc:299] Negotiation successful
0926 15:02:56.920016 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1054,"thread_start_us":105,"threads_started":1}
I20260926 15:02:56.921355 23085 heartbeater.cc:347] Connected to a master server at 127.8.102.62:45439
I20260926 15:02:56.921451 23085 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:56.921618 23085 heartbeater.cc:511] Master 127.8.102.62:45439 requested a full tablet report, sending...
I20260926 15:02:56.922392 22897 ts_manager.cc:194] Registered new tserver with Master: 7a452967a822457f83ef29ec496c8f60 (127.8.102.1:34853)
I20260926 15:02:56.923717 22897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:35997
I20260926 15:02:56.924611 23085 heartbeater.cc:503] Master 127.8.102.62:45439 was elected leader, sending a full tablet report...
Sep 26 15:02:56 dist-test-slave-jnhs krb5kdc[22317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434976, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for kudu/127.8.102.1@KRBTEST.COM
I20260926 15:02:56.933794 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.923317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43078 (local address 127.8.102.1:34853)
0926 15:02:56.923680 (+ 363us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:56.923684 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:56.923692 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:02:56.923723 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:56.923724 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:56.923737 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:56.923892 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:56.925264 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.926503 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.927647 (+ 1144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.928077 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.928159 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:56.930528 (+ 2369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:56.930536 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:56.930536 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:56.930544 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:56.932348 (+ 1804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:56.932800 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:56.932804 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:56.932804 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:56.932889 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:56.933122 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:56.933124 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:56.933124 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:56.933289 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:56.933387 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:56.933587 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 15:02:56.933687 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":66,"threads_started":1}
I20260926 15:02:56.933802 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:56.923073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:34853 (local address 127.0.0.1:43078)
0926 15:02:56.923468 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:56.923502 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:56.923564 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:56.924033 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:56.924036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:56.925063 (+ 1027us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:56.925120 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:56.925123 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.926655 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.926657 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:56.927524 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:56.927526 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:56.928168 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:56.928169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:56.928246 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:56.928853 (+ 607us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:56.928858 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:56.930415 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:56.932448 (+ 2033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:56.932455 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:56.932457 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:56.932687 (+ 230us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:56.932968 (+ 281us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:56.932969 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:56.932970 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:56.933042 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:56.933367 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:56.933373 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:56.933432 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 15:02:56.933542 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 15:02:56.933682 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":92,"threads_started":1}
I20260926 15:02:56.934329 23019 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ec2dfb00a94e7aa6433fc419670c09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:02:56 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:43078
I20260926 15:02:56.934959 23094 tablet_replica.cc:330] stopping tablet replica
I20260926 15:02:56.935081 23094 raft_consensus.cc:2229] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:02:56.935155 23094 raft_consensus.cc:2258] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:02:56.935467 23094 ts_tablet_manager.cc:1905] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:02:56.937865 23094 ts_tablet_manager.cc:1918] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:02:56.937948 23094 log.cc:1196] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/wal/wals/35ec2dfb00a94e7aa6433fc419670c09
I20260926 15:02:56.938186 23094 ts_tablet_manager.cc:1939] T 35ec2dfb00a94e7aa6433fc419670c09 P 7a452967a822457f83ef29ec496c8f60: Deleting consensus metadata
I20260926 15:02:56.938788 22883 catalog_manager.cc:4841] TS 7a452967a822457f83ef29ec496c8f60 (127.8.102.1:34853): tablet 35ec2dfb00a94e7aa6433fc419670c09 (table test-table [id=3f9853eb26c741abb71ff1f9619653f5]) successfully deleted
W20260926 15:02:56.989156 23087 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:56.989367 23087 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:56.989392 23087 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:56.990491 23087 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:56.990530 23087 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:56.990563 23087 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:02:56.992023 23087 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:36209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=33135
--tserver_master_addrs=127.8.102.62:45439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23087
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:56.992214 23087 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:56.992456 23087 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:56.997601 23100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:56.997680 23087 server_base.cc:1034] running on GCE node
W20260926 15:02:56.997591 23102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:56.997594 23099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:56.998067 23087 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:56.998381 23087 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:56.999536 23087 hybrid_clock.cc:648] HybridClock initialized: now 1790434976999525 us; error 34 us; skew 500 ppm
Sep 26 15:02:56 dist-test-slave-jnhs krb5kdc[22317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434976, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:57.002235 23087 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:02:57.002825 23087 webserver.cc:460] Webserver started at http://127.8.102.2:33135/ using document root <none> and password file <none>
I20260926 15:02:57.003065 23087 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:57.003159 23087 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:02:57.004645 23087 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:02:57.005406 23109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:57.005656 23087 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:57.005762 23087 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "43ca9df1eb32403fad905322acf736ca"
format_stamp: "Formatted at 2026-09-26 15:02:56 on dist-test-slave-jnhs"
server_key: "38fca355a3c10d9d276fee24d98a9c9a"
server_key_iv: "5a4fbddc1a9d1cedbe6e0e7aa1329c07"
server_key_version: "encryptionkey@0"
I20260926 15:02:57.006145 23087 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:57.016093 23087 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:57.018149 23087 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:57.018332 23087 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:57.018575 23087 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:02:57.019050 23116 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 15:02:57.020546 23087 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 15:02:57.020624 23087 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 15:02:57.020675 23087 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 15:02:57.021387 23087 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 15:02:57.021445 23087 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 15:02:57.021591 23116 tablet_bootstrap.cc:492] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca: Bootstrap starting.
I20260926 15:02:57.029062 23087 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:36209
I20260926 15:02:57.029173 23223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:36209 every 8 connection(s)
I20260926 15:02:57.029559 23087 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:02:57.033070 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 23087
I20260926 15:02:57.033216 23116 log.cc:824] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca: Log is configured to *not* fsync() on all Append() calls
I20260926 15:02:57.033412 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:41671
--local_ip_for_outbound_sockets=127.8.102.3
--tserver_master_addrs=127.8.102.62:45439
--webserver_port=36329
--webserver_interface=127.8.102.3
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:02:57.037071 23116 tablet_bootstrap.cc:492] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:02:57.037431 23116 tablet_bootstrap.cc:492] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca: Bootstrap complete.
Sep 26 15:02:57 dist-test-slave-jnhs krb5kdc[22317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434976, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:02:57.038552 23116 ts_tablet_manager.cc:1397] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.012s
I20260926 15:02:57.039871 23116 raft_consensus.cc:348] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } }
I20260926 15:02:57.040203 23116 raft_consensus.cc:729] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ca9df1eb32403fad905322acf736ca, State: Initialized, Role: FOLLOWER
I20260926 15:02:57.040388 23116 consensus_queue.cc:260] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } }
I20260926 15:02:57.040724 23116 ts_tablet_manager.cc:1428] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca: Time spent starting tablet: real 0.002s user 0.003s sys 0.003s
I20260926 15:02:57.043093 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.031355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.2:34125)
0926 15:02:57.031579 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:57.031618 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:57.032198 (+ 580us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:57.032451 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:57.032454 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:57.032732 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:57.032942 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:57.032947 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.034032 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.034033 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:57.035348 (+ 1315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:57.035351 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.035872 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.035874 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:57.036008 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:57.036623 (+ 615us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:57.036634 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:57.038786 (+ 2152us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:57.041314 (+ 2528us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:57.041321 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:57.041323 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:57.041664 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:57.041934 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:57.041934 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:57.041935 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:57.042068 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:57.042446 (+ 378us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:57.042451 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:57.042600 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 15:02:57.042774 (+ 174us) client_negotiation.cc:240] Negotiation successful
0926 15:02:57.042920 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":127,"threads_started":1}
I20260926 15:02:57.043128 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.031383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:34125 (local address 127.8.102.62:45439)
0926 15:02:57.031453 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:57.031456 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:57.031730 (+ 274us) server_negotiation.cc:407] Connection header received
0926 15:02:57.032318 (+ 588us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:57.032319 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:57.032332 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:57.032383 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:57.033045 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.033885 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.035476 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.035714 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.035840 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:57.038915 (+ 3075us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:57.038917 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:57.038917 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:57.038928 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:57.041193 (+ 2265us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:57.041769 (+ 576us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:57.041772 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:57.041772 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:57.041851 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:57.042176 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:57.042177 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:57.042177 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:57.042364 (+ 187us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:57.042451 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:57.042827 (+ 376us) server_negotiation.cc:299] Negotiation successful
0926 15:02:57.042954 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 15:02:57.043672 23224 heartbeater.cc:347] Connected to a master server at 127.8.102.62:45439
I20260926 15:02:57.043805 23224 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:57.044027 23224 heartbeater.cc:511] Master 127.8.102.62:45439 requested a full tablet report, sending...
I20260926 15:02:57.044796 22897 ts_manager.cc:194] Registered new tserver with Master: 43ca9df1eb32403fad905322acf736ca (127.8.102.2:36209)
I20260926 15:02:57.045403 22897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:34125
I20260926 15:02:57.046197 23224 heartbeater.cc:503] Master 127.8.102.62:45439 was elected leader, sending a full tablet report...
Sep 26 15:02:57 dist-test-slave-jnhs krb5kdc[22317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434976, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for kudu/127.8.102.2@KRBTEST.COM
I20260926 15:02:57.053388 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.045157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:36209 (local address 127.0.0.1:59108)
0926 15:02:57.045296 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:57.045309 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:57.045359 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:57.045680 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:57.045681 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:57.045886 (+ 205us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:57.045945 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:57.045948 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.046971 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.046971 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:57.047908 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:57.047910 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.048431 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.048432 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:57.048506 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:57.048947 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:57.048952 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:57.050351 (+ 1399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:57.052238 (+ 1887us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:57.052240 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:57.052241 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:57.052425 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:57.052657 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:57.052658 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:57.052658 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:57.052724 (+ 66us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:57.053027 (+ 303us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:57.053028 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:57.053076 (+ 48us) client_negotiation.cc:769] Sending connection context
0926 15:02:57.053165 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 15:02:57.053281 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115}
I20260926 15:02:57.053408 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.045307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59108 (local address 127.8.102.2:36209)
0926 15:02:57.045478 (+ 171us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:57.045483 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:57.045495 (+ 12us) server_negotiation.cc:407] Connection header received
0926 15:02:57.045535 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:57.045537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:57.045560 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:57.045673 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:57.046038 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.046888 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.048013 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.048347 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.048413 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:57.050450 (+ 2037us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:57.050459 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:57.050460 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:57.050474 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:57.052119 (+ 1645us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:57.052521 (+ 402us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:57.052526 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:57.052526 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:57.052585 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:57.052790 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:57.052791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:57.052791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:57.052945 (+ 154us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:57.053038 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:57.053202 (+ 164us) server_negotiation.cc:299] Negotiation successful
0926 15:02:57.053296 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":86,"threads_started":1}
I20260926 15:02:57.054112 23158 tablet_service.cc:1514] Processing DeleteTablet for tablet 35ec2dfb00a94e7aa6433fc419670c09 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:02:56 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:59108
I20260926 15:02:57.054765 23231 tablet_replica.cc:330] stopping tablet replica
I20260926 15:02:57.054900 23231 raft_consensus.cc:2229] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:02:57.054980 23231 raft_consensus.cc:2258] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:02:57.055341 23231 ts_tablet_manager.cc:1905] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:02:57.058223 23231 ts_tablet_manager.cc:1918] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:02:57.058312 23231 log.cc:1196] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/35ec2dfb00a94e7aa6433fc419670c09
I20260926 15:02:57.058559 23231 ts_tablet_manager.cc:1939] T 35ec2dfb00a94e7aa6433fc419670c09 P 43ca9df1eb32403fad905322acf736ca: Deleting consensus metadata
I20260926 15:02:57.059508 22884 catalog_manager.cc:4841] TS 43ca9df1eb32403fad905322acf736ca (127.8.102.2:36209): tablet 35ec2dfb00a94e7aa6433fc419670c09 (table test-table [id=3f9853eb26c741abb71ff1f9619653f5]) successfully deleted
W20260926 15:02:57.126400 23228 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:57.126673 23228 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:57.126701 23228 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:57.127868 23228 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:57.127907 23228 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:57.127935 23228 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:02:57.129401 23228 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:41671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=36329
--tserver_master_addrs=127.8.102.62:45439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23228
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:57.129606 23228 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:57.129802 23228 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:57.135253 23239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:57.135253 23236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:57.135253 23228 server_base.cc:1034] running on GCE node
W20260926 15:02:57.135270 23237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:57.135823 23228 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:57.136096 23228 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:57.137271 23228 hybrid_clock.cc:648] HybridClock initialized: now 1790434977137255 us; error 68 us; skew 500 ppm
Sep 26 15:02:57 dist-test-slave-jnhs krb5kdc[22317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434977, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:57.139937 23228 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:02:57.140333 23228 webserver.cc:460] Webserver started at http://127.8.102.3:36329/ using document root <none> and password file <none>
I20260926 15:02:57.140542 23228 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:57.140595 23228 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:02:57.141928 23228 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:57.142553 23246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:57.142732 23228 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:57.142825 23228 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "0eabcd36a11845d5b9bc40918437fa3b"
format_stamp: "Formatted at 2026-09-26 15:02:56 on dist-test-slave-jnhs"
server_key: "223b5a3b280223492d3b75e87232a83d"
server_key_iv: "5215461b21f1207046b2c40b5770ef7d"
server_key_version: "encryptionkey@0"
I20260926 15:02:57.143170 23228 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:57.189796 23228 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:57.192209 23228 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:57.192333 23228 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:57.192549 23228 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:02:57.192812 23228 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:02:57.192843 23228 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:57.192866 23228 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:02:57.192883 23228 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:57.199084 23228 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:41671
I20260926 15:02:57.199185 23359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:41671 every 8 connection(s)
I20260926 15:02:57.199540 23228 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:02:57.200299 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 23228
Sep 26 15:02:57 dist-test-slave-jnhs krb5kdc[22317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434977, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:02:57.207940 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.200712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45242 (local address 127.8.102.62:45439)
0926 15:02:57.200846 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:57.200849 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:57.200887 (+ 38us) server_negotiation.cc:407] Connection header received
0926 15:02:57.200978 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:57.200978 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:57.200990 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:57.201072 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:57.201665 (+ 593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.202181 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.203001 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.203263 (+ 262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.203337 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:57.204634 (+ 1297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:57.204635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:57.204635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:57.204643 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:57.206373 (+ 1730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:57.206842 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:57.206844 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:57.206844 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:57.206906 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:57.207372 (+ 466us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:57.207374 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:57.207374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:57.207551 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:57.207751 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:57.207794 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 15:02:57.207868 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":63,"threads_started":1}
I20260926 15:02:57.210244 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.200356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:39101 (local address 127.8.102.62:45439)
0926 15:02:57.200413 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:57.200415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:57.200949 (+ 534us) server_negotiation.cc:407] Connection header received
0926 15:02:57.201464 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:57.201465 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:57.201476 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:57.201523 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:57.202466 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.203016 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.204192 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.204400 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.204484 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:57.206979 (+ 2495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:57.206980 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:57.206981 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:57.206988 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:57.208928 (+ 1940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:57.209336 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:57.209338 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:57.209338 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:57.209407 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:57.209632 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:57.209632 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:57.209633 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:57.209734 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:57.209792 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:57.210060 (+ 268us) server_negotiation.cc:299] Negotiation successful
0926 15:02:57.210180 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 15:02:57.210362 23362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.200361 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.3:39101)
0926 15:02:57.200826 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:57.200863 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:57.201366 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:57.201605 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:57.201608 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:57.202173 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:57.202362 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:57.202366 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.203118 (+ 752us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.203119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:57.204092 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:57.204094 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.204497 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.204497 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:57.204624 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:57.205123 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:57.205134 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:57.206892 (+ 1758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:57.209004 (+ 2112us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:57.209011 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:57.209013 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:57.209244 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:57.209475 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:57.209475 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:57.209476 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:57.209576 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:57.209800 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:57.209805 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:57.209950 (+ 145us) client_negotiation.cc:769] Sending connection context
0926 15:02:57.210035 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 15:02:57.210161 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":95,"threads_started":1}
I20260926 15:02:57.210903 23360 heartbeater.cc:347] Connected to a master server at 127.8.102.62:45439
I20260926 15:02:57.211016 23360 heartbeater.cc:464] Registering TS with master...
I20260926 15:02:57.211211 23360 heartbeater.cc:511] Master 127.8.102.62:45439 requested a full tablet report, sending...
I20260926 15:02:57.211594 22897 ts_manager.cc:194] Registered new tserver with Master: 0eabcd36a11845d5b9bc40918437fa3b (127.8.102.3:41671)
I20260926 15:02:57.211933 22897 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:39101
I20260926 15:02:57.219998 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:02:57.223259 23090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.220396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45254 (local address 127.8.102.62:45439)
0926 15:02:57.220498 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:57.220500 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:57.220591 (+ 91us) server_negotiation.cc:407] Connection header received
0926 15:02:57.220662 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:57.220662 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:57.220696 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:57.220764 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:02:57.220968 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.221572 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.222381 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.222583 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.222747 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:57.222858 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:02:57.222966 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:02:57.223034 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:57.223066 (+ 32us) server_negotiation.cc:299] Negotiation successful
0926 15:02:57.223088 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260926 15:02:57.223579 22897 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:45254:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:02:57.225394 22897 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:02:57.228052 23019 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5ce8df704143c287f153891cdde3ad (DEFAULT_TABLE table=test-table [id=47e051f7c8624f55930ebfaf923d110e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:02:57.228042 23158 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5ce8df704143c287f153891cdde3ad (DEFAULT_TABLE table=test-table [id=47e051f7c8624f55930ebfaf923d110e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:02:57.228241 23158 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5ce8df704143c287f153891cdde3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:02:57.228241 23019 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5ce8df704143c287f153891cdde3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:02:57.229812 23116 tablet_bootstrap.cc:492] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca: Bootstrap starting.
I20260926 15:02:57.229812 22977 tablet_bootstrap.cc:492] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60: Bootstrap starting.
I20260926 15:02:57.230165 23116 tablet_bootstrap.cc:654] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca: Neither blocks nor log segments found. Creating new log.
I20260926 15:02:57.230165 22977 tablet_bootstrap.cc:654] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60: Neither blocks nor log segments found. Creating new log.
I20260926 15:02:57.230770 23116 tablet_bootstrap.cc:492] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca: No bootstrap required, opened a new log
I20260926 15:02:57.230834 23116 ts_tablet_manager.cc:1397] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260926 15:02:57.230888 22977 tablet_bootstrap.cc:492] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60: No bootstrap required, opened a new log
I20260926 15:02:57.230940 22977 ts_tablet_manager.cc:1397] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:57.231257 23116 raft_consensus.cc:348] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } }
I20260926 15:02:57.231272 22977 raft_consensus.cc:348] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } }
I20260926 15:02:57.231345 22977 raft_consensus.cc:374] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:02:57.231348 23116 raft_consensus.cc:374] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:02:57.231388 22977 raft_consensus.cc:729] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a452967a822457f83ef29ec496c8f60, State: Initialized, Role: FOLLOWER
I20260926 15:02:57.231395 23116 raft_consensus.cc:729] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43ca9df1eb32403fad905322acf736ca, State: Initialized, Role: FOLLOWER
I20260926 15:02:57.231442 23116 consensus_queue.cc:260] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } }
I20260926 15:02:57.231433 22977 consensus_queue.cc:260] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } }
I20260926 15:02:57.231534 23116 ts_tablet_manager.cc:1428] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca: Time spent starting tablet: real 0.001s user 0.000s sys 0.001s
I20260926 15:02:57.231540 22977 ts_tablet_manager.cc:1428] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:57.233747 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.227747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:41671 (local address 127.0.0.1:54058)
0926 15:02:57.227798 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:57.227811 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:57.227852 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:57.230123 (+ 2271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:57.230124 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:57.230126 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:02:57.230175 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:57.230177 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.231744 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.231745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:57.232813 (+ 1068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:57.232816 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.233522 (+ 706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.233523 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:57.233582 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:57.233598 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 15:02:57.233651 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 15:02:57.233687 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18}
I20260926 15:02:57.233770 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.227858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54058 (local address 127.8.102.3:41671)
0926 15:02:57.229926 (+ 2068us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:57.229932 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:57.229943 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:02:57.229985 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:57.229987 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:57.230005 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:57.230117 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:02:57.230232 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.231414 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.232894 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.233425 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.233489 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:57.233533 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:57.233676 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 15:02:57.233708 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2027,"thread_start_us":110,"threads_started":1}
I20260926 15:02:57.234463 23294 tablet_service.cc:1467] Processing CreateTablet for tablet 1a5ce8df704143c287f153891cdde3ad (DEFAULT_TABLE table=test-table [id=47e051f7c8624f55930ebfaf923d110e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:02:57.234899 23294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a5ce8df704143c287f153891cdde3ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:02:57.237591 23369 tablet_bootstrap.cc:492] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b: Bootstrap starting.
I20260926 15:02:57.238224 23369 tablet_bootstrap.cc:654] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b: Neither blocks nor log segments found. Creating new log.
I20260926 15:02:57.238804 23369 log.cc:824] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b: Log is configured to *not* fsync() on all Append() calls
I20260926 15:02:57.239554 23369 tablet_bootstrap.cc:492] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b: No bootstrap required, opened a new log
I20260926 15:02:57.239641 23369 ts_tablet_manager.cc:1397] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 15:02:57.240998 23369 raft_consensus.cc:348] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } }
I20260926 15:02:57.241199 23369 raft_consensus.cc:374] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:02:57.241274 23369 raft_consensus.cc:729] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eabcd36a11845d5b9bc40918437fa3b, State: Initialized, Role: FOLLOWER
I20260926 15:02:57.241403 23369 consensus_queue.cc:260] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } }
I20260926 15:02:57.241698 23369 ts_tablet_manager.cc:1428] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 15:02:57.241732 23360 heartbeater.cc:503] Master 127.8.102.62:45439 was elected leader, sending a full tablet report...
W20260926 15:02:57.280217 23225 tablet.cc:2369] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:02:57.396312 23229 raft_consensus.cc:482] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:02:57.396517 23229 raft_consensus.cc:504] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } }
I20260926 15:02:57.396886 23229 leader_election.cc:288] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0eabcd36a11845d5b9bc40918437fa3b (127.8.102.3:41671), 7a452967a822457f83ef29ec496c8f60 (127.8.102.1:34853)
W20260926 15:02:57.399740 23086 tablet.cc:2369] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:02:57.401074 23093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.397582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:43865 (local address 127.8.102.1:34853)
0926 15:02:57.397638 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:57.397640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:57.397762 (+ 122us) server_negotiation.cc:407] Connection header received
0926 15:02:57.397813 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:57.397814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:57.397863 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:57.397930 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:02:57.398087 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.398829 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.400097 (+ 1268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.400701 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.400766 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:57.400845 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:57.400966 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 15:02:57.401006 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 15:02:57.401074 23366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.397522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:52737 (local address 127.8.102.3:41671)
0926 15:02:57.397571 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:57.397573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:57.397581 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:02:57.397610 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:57.397611 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:57.397623 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:57.397681 (+ 58us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:02:57.397883 (+ 202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.398630 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.400098 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.400663 (+ 565us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.400766 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:57.400779 (+ 13us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:57.400966 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 15:02:57.401006 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 15:02:57.401099 23226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.397262 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:41671 (local address 127.8.102.2:52737)
0926 15:02:57.397327 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:57.397340 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:57.397400 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:57.397717 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:57.397718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:57.397721 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:02:57.397806 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:57.397808 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.398724 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.398724 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:57.399991 (+ 1267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:57.399994 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.400777 (+ 783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.400778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:57.400860 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:57.400886 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 15:02:57.400948 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 15:02:57.400995 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41,"mutex_wait_us":30}
I20260926 15:02:57.401113 23372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.397404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:34853 (local address 127.8.102.2:43865)
0926 15:02:57.397644 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:57.397656 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:57.397752 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:57.397951 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:57.397951 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:57.397953 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:02:57.398012 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:57.398013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.398903 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.398904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:57.400001 (+ 1097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:57.400003 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.400801 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.400802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:57.400860 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:57.400886 (+ 26us) client_negotiation.cc:769] Sending connection context
0926 15:02:57.400948 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 15:02:57.400995 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":214,"mutex_wait_us":4,"thread_start_us":140,"threads_started":1}
I20260926 15:02:57.401460 23039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5ce8df704143c287f153891cdde3ad" candidate_uuid: "43ca9df1eb32403fad905322acf736ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a452967a822457f83ef29ec496c8f60" is_pre_election: true
I20260926 15:02:57.401504 23314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5ce8df704143c287f153891cdde3ad" candidate_uuid: "43ca9df1eb32403fad905322acf736ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eabcd36a11845d5b9bc40918437fa3b" is_pre_election: true
I20260926 15:02:57.401600 23039 raft_consensus.cc:2454] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43ca9df1eb32403fad905322acf736ca in term 0.
I20260926 15:02:57.401683 23314 raft_consensus.cc:2454] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 43ca9df1eb32403fad905322acf736ca in term 0.
I20260926 15:02:57.401861 23112 leader_election.cc:302] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 43ca9df1eb32403fad905322acf736ca, 7a452967a822457f83ef29ec496c8f60; no voters:
I20260926 15:02:57.401984 23229 raft_consensus.cc:2781] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:02:57.402038 23229 raft_consensus.cc:482] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:02:57.402062 23229 raft_consensus.cc:3037] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:02:57.402801 23229 raft_consensus.cc:504] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } }
I20260926 15:02:57.402936 23229 leader_election.cc:288] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [CANDIDATE]: Term 1 election: Requested vote from peers 0eabcd36a11845d5b9bc40918437fa3b (127.8.102.3:41671), 7a452967a822457f83ef29ec496c8f60 (127.8.102.1:34853)
I20260926 15:02:57.403136 23314 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5ce8df704143c287f153891cdde3ad" candidate_uuid: "43ca9df1eb32403fad905322acf736ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0eabcd36a11845d5b9bc40918437fa3b"
I20260926 15:02:57.403136 23039 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a5ce8df704143c287f153891cdde3ad" candidate_uuid: "43ca9df1eb32403fad905322acf736ca" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7a452967a822457f83ef29ec496c8f60"
I20260926 15:02:57.403224 23314 raft_consensus.cc:3037] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:02:57.403255 23039 raft_consensus.cc:3037] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:02:57.404084 23314 raft_consensus.cc:2454] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43ca9df1eb32403fad905322acf736ca in term 1.
I20260926 15:02:57.404084 23039 raft_consensus.cc:2454] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 43ca9df1eb32403fad905322acf736ca in term 1.
I20260926 15:02:57.404290 23112 leader_election.cc:302] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0eabcd36a11845d5b9bc40918437fa3b, 43ca9df1eb32403fad905322acf736ca; no voters:
I20260926 15:02:57.404399 23229 raft_consensus.cc:2781] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:02:57.404542 23229 raft_consensus.cc:686] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [term 1 LEADER]: Becoming Leader. State: Replica: 43ca9df1eb32403fad905322acf736ca, State: Running, Role: LEADER
I20260926 15:02:57.404598 23229 consensus_queue.cc:237] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } }
I20260926 15:02:57.405447 22897 catalog_manager.cc:5495] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca reported cstate change: term changed from 0 to 1, leader changed from <none> to 43ca9df1eb32403fad905322acf736ca (127.8.102.2). New cstate: current_term: 1 leader_uuid: "43ca9df1eb32403fad905322acf736ca" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } health_report { overall_health: UNKNOWN } } }
I20260926 15:02:57.416206 23230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.413476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59114 (local address 127.8.102.2:36209)
0926 15:02:57.413534 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:57.413536 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:57.413546 (+ 10us) server_negotiation.cc:407] Connection header received
0926 15:02:57.413626 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:57.413631 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:57.413704 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:57.413790 (+ 86us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:02:57.413925 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.414483 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.415404 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:57.415604 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:57.415696 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:57.415860 (+ 164us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:02:57.415962 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:02:57.416060 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:57.416100 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 15:02:57.416122 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 15:02:57.417663 23314 raft_consensus.cc:1261] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b [term 1 FOLLOWER]: Refusing update from remote peer 43ca9df1eb32403fad905322acf736ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:02:57.417682 23039 raft_consensus.cc:1261] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Refusing update from remote peer 43ca9df1eb32403fad905322acf736ca: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:02:57.418032 23229 consensus_queue.cc:1035] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:02:57.418174 23373 consensus_queue.cc:1035] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [LEADER]: Connected to new peer: Peer: permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:02:57.420039 23375 mvcc.cc:205] Tried to move back new op lower bound from 7333621667500269568 to 7333621667449909248. Current Snapshot: MvccSnapshot[applied={T|T < 7333621667500269568}]
I20260926 15:02:57.420665 23377 mvcc.cc:205] Tried to move back new op lower bound from 7333621667500269568 to 7333621667449909248. Current Snapshot: MvccSnapshot[applied={T|T < 7333621667500269568}]
I20260926 15:02:57.423127 22895 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:45254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:02:57.423213 22895 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:45254:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:02:57.423959 22895 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3: Sending DeleteTablet for 3 replicas of tablet 1a5ce8df704143c287f153891cdde3ad
I20260926 15:02:57.424309 23294 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a5ce8df704143c287f153891cdde3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:02:57 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:54058
I20260926 15:02:57.424325 23019 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a5ce8df704143c287f153891cdde3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:02:57 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:43078
I20260926 15:02:57.424324 23158 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a5ce8df704143c287f153891cdde3ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:02:57 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:59108
I20260926 15:02:57.424413 23094 tablet_replica.cc:330] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60: stopping tablet replica
I20260926 15:02:57.424413 23231 tablet_replica.cc:330] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca: stopping tablet replica
I20260926 15:02:57.424454 23231 raft_consensus.cc:2229] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:02:57.424458 23094 raft_consensus.cc:2229] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:02:57.424496 23094 raft_consensus.cc:2258] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:02:57.424548 23231 raft_consensus.cc:2258] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:02:57.424707 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 22936
I20260926 15:02:57.424844 23094 ts_tablet_manager.cc:1905] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:02:57.424841 23231 ts_tablet_manager.cc:1905] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:02:57.426186 23385 tablet_replica.cc:330] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b: stopping tablet replica
I20260926 15:02:57.426265 23385 raft_consensus.cc:2229] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:02:57.426337 23385 raft_consensus.cc:2258] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:02:57.426769 23231 ts_tablet_manager.cc:1918] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:02:57.426819 23231 log.cc:1196] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/1a5ce8df704143c287f153891cdde3ad
I20260926 15:02:57.426849 23385 ts_tablet_manager.cc:1905] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:02:57.427032 23231 ts_tablet_manager.cc:1939] T 1a5ce8df704143c287f153891cdde3ad P 43ca9df1eb32403fad905322acf736ca: Deleting consensus metadata
I20260926 15:02:57.427361 22884 catalog_manager.cc:4841] TS 43ca9df1eb32403fad905322acf736ca (127.8.102.2:36209): tablet 1a5ce8df704143c287f153891cdde3ad (table test-table [id=47e051f7c8624f55930ebfaf923d110e]) successfully deleted
I20260926 15:02:57.428380 23385 ts_tablet_manager.cc:1918] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:02:57.428460 23385 log.cc:1196] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/1a5ce8df704143c287f153891cdde3ad
I20260926 15:02:57.428762 23385 ts_tablet_manager.cc:1939] T 1a5ce8df704143c287f153891cdde3ad P 0eabcd36a11845d5b9bc40918437fa3b: Deleting consensus metadata
I20260926 15:02:57.429143 22883 catalog_manager.cc:4841] TS 0eabcd36a11845d5b9bc40918437fa3b (127.8.102.3:41671): tablet 1a5ce8df704143c287f153891cdde3ad (table test-table [id=47e051f7c8624f55930ebfaf923d110e]) successfully deleted
W20260926 15:02:57.432461 22883 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.102.1:34853 (error 108)
I20260926 15:02:57.433044 23092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.432910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:34853 (local address 127.0.0.1:43092)
0926 15:02:57.432965 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:57.433003 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.1:34853: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 15:02:57.433097 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 23087
I20260926 15:02:57.437530 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 23228
W20260926 15:02:57.437510 22883 catalog_manager.cc:4584] TS 7a452967a822457f83ef29ec496c8f60 (127.8.102.1:34853): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1a5ce8df704143c287f153891cdde3ad: Network error: Client connection negotiation failed: client connection to 127.8.102.1:34853: connect: Connection refused (error 111)
I20260926 15:02:57.442173 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 22864
I20260926 15:02:57.447640 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:45439
--webserver_interface=127.8.102.62
--webserver_port=39379
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:45439
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:02:57.529457 23387 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:57.529678 23387 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:57.529702 23387 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:57.530912 23387 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:02:57.530956 23387 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:57.530970 23387 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:57.530983 23387 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:02:57.530995 23387 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:02:57.532533 23387 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:45439
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:45439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=39379
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23387
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:57.532806 23387 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:57.533057 23387 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:57.538372 23392 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:57.538367 23393 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:57.538368 23395 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:57.538708 23387 server_base.cc:1034] running on GCE node
I20260926 15:02:57.538975 23387 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:57.539296 23387 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:57.540478 23387 hybrid_clock.cc:648] HybridClock initialized: now 1790434977540455 us; error 41 us; skew 500 ppm
Sep 26 15:02:57 dist-test-slave-jnhs krb5kdc[22317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434977, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:57.543073 23387 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:02:57.543628 23387 webserver.cc:460] Webserver started at http://127.8.102.62:39379/ using document root <none> and password file <none>
I20260926 15:02:57.543867 23387 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:57.543939 23387 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:02:57.545356 23387 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:57.546061 23402 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:57.546288 23387 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:57.546396 23387 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "a008a79af4574e19aee1b8219d5f1ed3"
format_stamp: "Formatted at 2026-09-26 15:02:55 on dist-test-slave-jnhs"
server_key: "7041b229940adbcd4ead3f6d8550aaf3"
server_key_iv: "4b19a959167e0c256b1fa2323600e243"
server_key_version: "encryptionkey@0"
I20260926 15:02:57.546808 23387 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:57.581416 23387 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:57.583678 23387 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:57.583858 23387 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:57.588034 23387 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:45439
I20260926 15:02:57.588083 23454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:45439 every 8 connection(s)
I20260926 15:02:57.588516 23387 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:02:57.590178 23455 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:02:57.590965 23455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3: Bootstrap starting.
I20260926 15:02:57.593137 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 23387
I20260926 15:02:57.593461 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:34853
--local_ip_for_outbound_sockets=127.8.102.1
--tserver_master_addrs=127.8.102.62:45439
--webserver_port=44749
--webserver_interface=127.8.102.1
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:02:57.600016 23455 log.cc:824] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3: Log is configured to *not* fsync() on all Append() calls
I20260926 15:02:57.602324 23455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:02:57.602716 23455 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3: Bootstrap complete.
I20260926 15:02:57.604998 23455 raft_consensus.cc:348] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a008a79af4574e19aee1b8219d5f1ed3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45439 } }
I20260926 15:02:57.605233 23455 raft_consensus.cc:729] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: a008a79af4574e19aee1b8219d5f1ed3, State: Initialized, Role: FOLLOWER
I20260926 15:02:57.605415 23455 consensus_queue.cc:260] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a008a79af4574e19aee1b8219d5f1ed3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45439 } }
I20260926 15:02:57.605477 23455 raft_consensus.cc:388] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:02:57.605541 23455 raft_consensus.cc:482] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:02:57.605592 23455 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 2 FOLLOWER]: Advancing to term 3
I20260926 15:02:57.606612 23455 raft_consensus.cc:504] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a008a79af4574e19aee1b8219d5f1ed3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45439 } }
I20260926 15:02:57.606782 23455 leader_election.cc:302] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a008a79af4574e19aee1b8219d5f1ed3; no voters:
I20260926 15:02:57.606999 23455 leader_election.cc:288] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 15:02:57.607043 23459 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 3 FOLLOWER]: Leader election won for term 3
I20260926 15:02:57.607239 23459 raft_consensus.cc:686] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [term 3 LEADER]: Becoming Leader. State: Replica: a008a79af4574e19aee1b8219d5f1ed3, State: Running, Role: LEADER
I20260926 15:02:57.607302 23455 sys_catalog.cc:564] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:02:57.607342 23459 consensus_queue.cc:237] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a008a79af4574e19aee1b8219d5f1ed3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45439 } }
I20260926 15:02:57.607640 23459 sys_catalog.cc:455] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a008a79af4574e19aee1b8219d5f1ed3. Latest consensus state: current_term: 3 leader_uuid: "a008a79af4574e19aee1b8219d5f1ed3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a008a79af4574e19aee1b8219d5f1ed3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45439 } } }
I20260926 15:02:57.607770 23459 sys_catalog.cc:458] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [sys.catalog]: This master's current role is: LEADER
I20260926 15:02:57.607646 23460 sys_catalog.cc:455] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "a008a79af4574e19aee1b8219d5f1ed3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a008a79af4574e19aee1b8219d5f1ed3" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 45439 } } }
I20260926 15:02:57.607941 23460 sys_catalog.cc:458] T 00000000000000000000000000000000 P a008a79af4574e19aee1b8219d5f1ed3 [sys.catalog]: This master's current role is: LEADER
I20260926 15:02:57.607990 23466 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:02:57.608870 23466 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:02:57.608973 23466 catalog_manager.cc:1259] Loaded cluster ID: c7b7a7f3fbed451da33f3726329d8cda
I20260926 15:02:57.609009 23466 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260926 15:02:57.609223 23466 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fdf027bb9ad google::LogMessage::Fail() at ??:0
@ 0x7fdf027be88a google::LogMessage::SendToLog() at ??:0
@ 0x7fdf027bb4de google::LogMessage::Flush() at ??:0
@ 0x7fdf027bca19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fdf0a6dc0f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7fdf0a6e58b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fdf046890d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fdf0467e954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fdf054e26db start_thread at ??:0
@ 0x7fdf012b961f clone at ??:0
W20260926 15:02:57.691131 23457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:57.691355 23457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:57.691382 23457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:57.692636 23457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:57.692685 23457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:57.692715 23457 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:02:57.694306 23457 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:34853
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=44749
--tserver_master_addrs=127.8.102.62:45439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23457
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:57.694592 23457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:57.694909 23457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:57.700268 23491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:57.700222 23494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:57.700487 23457 server_base.cc:1034] running on GCE node
W20260926 15:02:57.700222 23492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:57.700795 23457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:57.701099 23457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:57.702311 23457 hybrid_clock.cc:648] HybridClock initialized: now 1790434977702294 us; error 31 us; skew 500 ppm
Sep 26 15:02:57 dist-test-slave-jnhs krb5kdc[22317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434977, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:57.705201 23457 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:02:57.705730 23457 webserver.cc:460] Webserver started at http://127.8.102.1:44749/ using document root <none> and password file <none>
I20260926 15:02:57.705967 23457 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:57.706036 23457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:02:57.707628 23457 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:57.708530 23501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:57.708770 23457 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:57.708868 23457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "7a452967a822457f83ef29ec496c8f60"
format_stamp: "Formatted at 2026-09-26 15:02:55 on dist-test-slave-jnhs"
server_key: "9fb97960cce0797b74fe48aefe3da002"
server_key_iv: "471b263716f8522cf0dc561a1449526f"
server_key_version: "encryptionkey@0"
I20260926 15:02:57.709260 23457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:57.744371 23457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:57.747015 23457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:57.747205 23457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:57.747499 23457 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:02:57.748099 23508 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 15:02:57.749711 23457 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 15:02:57.749775 23457 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 15:02:57.749814 23457 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 15:02:57.750751 23457 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 15:02:57.750798 23457 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:57.750883 23508 tablet_bootstrap.cc:492] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60: Bootstrap starting.
I20260926 15:02:57.752696 23508 log.cc:824] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60: Log is configured to *not* fsync() on all Append() calls
I20260926 15:02:57.755946 23508 tablet_bootstrap.cc:492] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:02:57.756284 23508 tablet_bootstrap.cc:492] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60: Bootstrap complete.
I20260926 15:02:57.756493 23508 ts_tablet_manager.cc:1397] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260926 15:02:57.757913 23457 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:34853
I20260926 15:02:57.758028 23616 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:34853 every 8 connection(s)
I20260926 15:02:57.758304 23457 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:02:57.758404 23508 raft_consensus.cc:348] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } }
I20260926 15:02:57.758922 23508 raft_consensus.cc:729] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a452967a822457f83ef29ec496c8f60, State: Initialized, Role: FOLLOWER
I20260926 15:02:57.759085 23508 consensus_queue.cc:260] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } }
I20260926 15:02:57.759335 23508 ts_tablet_manager.cc:1428] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:02:57.759589 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 23457
I20260926 15:02:57.759855 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:36209
--local_ip_for_outbound_sockets=127.8.102.2
--tserver_master_addrs=127.8.102.62:45439
--webserver_port=33135
--webserver_interface=127.8.102.2
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:02:57.841917 23619 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:02:57.759011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.1:44981)
0926 15:02:57.759182 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:57.759229 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:57.759386 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:57.841683 (+ 82297us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":77,"threads_started":1}
W20260926 15:02:57.842283 23617 heartbeater.cc:644] Failed to heartbeat to 127.8.102.62:45439 (0 consecutive failures): Network error: Failed to ping master at 127.8.102.62:45439: Client connection negotiation failed: client connection to 127.8.102.62:45439: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 15:02:57.842701 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.842525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.1:39241)
0926 15:02:57.842587 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:57.842672 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 15:02:57.843021 23619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.842924 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.1:36095)
0926 15:02:57.842971 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:57.842995 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
W20260926 15:02:57.843125 23617 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 15:02:57.844409 23621 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:57.844632 23621 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:57.844684 23621 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:57.845887 23621 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:57.845929 23621 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:57.845957 23621 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:02:57.847455 23621 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:36209
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=33135
--tserver_master_addrs=127.8.102.62:45439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23621
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:57.847661 23621 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:57.847842 23621 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:57.853031 23626 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:57.853034 23627 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:57.853233 23629 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:57.853235 23621 server_base.cc:1034] running on GCE node
I20260926 15:02:57.853580 23621 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:57.853844 23621 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:57.855063 23621 hybrid_clock.cc:648] HybridClock initialized: now 1790434977855050 us; error 32 us; skew 500 ppm
Sep 26 15:02:57 dist-test-slave-jnhs krb5kdc[22317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434977, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:57.857705 23621 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:02:57.858201 23621 webserver.cc:460] Webserver started at http://127.8.102.2:33135/ using document root <none> and password file <none>
I20260926 15:02:57.858397 23621 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:57.858438 23621 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:02:57.859856 23621 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:57.860639 23636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:57.860946 23621 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:57.861037 23621 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "43ca9df1eb32403fad905322acf736ca"
format_stamp: "Formatted at 2026-09-26 15:02:56 on dist-test-slave-jnhs"
server_key: "38fca355a3c10d9d276fee24d98a9c9a"
server_key_iv: "5a4fbddc1a9d1cedbe6e0e7aa1329c07"
server_key_version: "encryptionkey@0"
I20260926 15:02:57.861502 23621 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:57.876129 23621 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:57.878604 23621 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:57.878845 23621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:57.879109 23621 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:02:57.879449 23621 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:02:57.879489 23621 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:57.879513 23621 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:02:57.879555 23621 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:57.885948 23621 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:36209
I20260926 15:02:57.885993 23749 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:36209 every 8 connection(s)
I20260926 15:02:57.886351 23621 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:02:57.887478 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.887035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.2:43097)
0926 15:02:57.887200 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:57.887314 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":77,"threads_started":1}
W20260926 15:02:57.888002 23750 heartbeater.cc:644] Failed to heartbeat to 127.8.102.62:45439 (0 consecutive failures): Network error: Failed to ping master at 127.8.102.62:45439: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
I20260926 15:02:57.888383 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.888280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.2:53307)
0926 15:02:57.888332 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:57.888363 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 15:02:57.888653 23752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:57.888602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.2:49333)
0926 15:02:57.888623 (+ 21us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:57.888641 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":12}
W20260926 15:02:57.888778 23750 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 15:02:57.895563 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 23621
I20260926 15:02:57.895836 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:41671
--local_ip_for_outbound_sockets=127.8.102.3
--tserver_master_addrs=127.8.102.62:45439
--webserver_port=36329
--webserver_interface=127.8.102.3
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:43065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:02:57.979835 23753 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:02:57.980072 23753 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:02:57.980098 23753 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:02:57.981272 23753 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:02:57.981315 23753 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:02:57.981343 23753 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:02:57.982867 23753 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:43065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:41671
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=36329
--tserver_master_addrs=127.8.102.62:45439
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23753
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:02:57.983089 23753 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:02:57.983273 23753 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:02:57.988513 23759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:02:57.988528 23758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:57.988726 23753 server_base.cc:1034] running on GCE node
W20260926 15:02:57.988739 23761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:02:57.989009 23753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:02:57.989269 23753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:02:57.990418 23753 hybrid_clock.cc:648] HybridClock initialized: now 1790434977990397 us; error 38 us; skew 500 ppm
Sep 26 15:02:57 dist-test-slave-jnhs krb5kdc[22317](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434977, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:02:57.993054 23753 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:02:57.993610 23753 webserver.cc:460] Webserver started at http://127.8.102.3:36329/ using document root <none> and password file <none>
I20260926 15:02:57.993877 23753 fs_manager.cc:359] Metadata directory not provided
I20260926 15:02:57.993925 23753 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:02:57.995450 23753 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 15:02:57.996112 23768 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:57.996364 23753 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:02:57.996487 23753 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "0eabcd36a11845d5b9bc40918437fa3b"
format_stamp: "Formatted at 2026-09-26 15:02:56 on dist-test-slave-jnhs"
server_key: "223b5a3b280223492d3b75e87232a83d"
server_key_iv: "5215461b21f1207046b2c40b5770ef7d"
server_key_version: "encryptionkey@0"
I20260926 15:02:57.996871 23753 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:02:58.024801 23753 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:02:58.027626 23753 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:02:58.027812 23753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:02:58.028083 23753 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:02:58.028411 23753 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:02:58.028466 23753 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:58.028507 23753 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:02:58.028537 23753 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:02:58.034854 23753 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:41671
I20260926 15:02:58.034945 23881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:41671 every 8 connection(s)
I20260926 15:02:58.035349 23753 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:02:58.036525 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:58.036072 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.3:34111)
0926 15:02:58.036282 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:58.036381 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":67,"threads_started":1}
W20260926 15:02:58.036955 23882 heartbeater.cc:644] Failed to heartbeat to 127.8.102.62:45439 (0 consecutive failures): Network error: Failed to ping master at 127.8.102.62:45439: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
I20260926 15:02:58.037333 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:58.037208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.3:55731)
0926 15:02:58.037278 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:58.037311 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260926 15:02:58.037832 23884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:58.037739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.3:42599)
0926 15:02:58.037783 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:58.037809 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260926 15:02:58.038012 23882 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 15:02:58.041742 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 23753
I20260926 15:02:58.843844 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:58.843534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.1:46515)
0926 15:02:58.843727 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:58.843781 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":171,"thread_start_us":83,"threads_started":1}
I20260926 15:02:58.889477 23887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:58.889209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.2:57935)
0926 15:02:58.889374 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:58.889419 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":63,"threads_started":1}
I20260926 15:02:58.933905 23888 raft_consensus.cc:482] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:02:58.934048 23888 raft_consensus.cc:504] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } }
I20260926 15:02:58.934500 23888 leader_election.cc:288] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0eabcd36a11845d5b9bc40918437fa3b (127.8.102.3:41671), 43ca9df1eb32403fad905322acf736ca (127.8.102.2:36209)
Sep 26 15:02:58 dist-test-slave-jnhs krb5kdc[22317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434977, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.2@KRBTEST.COM
Sep 26 15:02:58 dist-test-slave-jnhs krb5kdc[22317](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434977, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.3@KRBTEST.COM
I20260926 15:02:58.946895 23891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:58.934899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:39729 (local address 127.8.102.2:36209)
0926 15:02:58.935083 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:58.935096 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:58.935137 (+ 41us) server_negotiation.cc:407] Connection header received
0926 15:02:58.935591 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:58.935593 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:58.935634 (+ 41us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:58.935796 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:58.936872 (+ 1076us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:58.938193 (+ 1321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:58.939342 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:58.939854 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:58.939978 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:58.942434 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:58.942438 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:58.942438 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:58.942451 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:58.944782 (+ 2331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:58.945560 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:58.945562 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:58.945562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:58.945645 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:58.945858 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:58.945858 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:58.945859 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:58.946112 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:58.946185 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:58.946682 (+ 497us) server_negotiation.cc:299] Negotiation successful
0926 15:02:58.946811 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":63,"threads_started":1}
I20260926 15:02:58.947021 23889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:58.934779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:36209 (local address 127.8.102.1:39729)
0926 15:02:58.934962 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:58.934972 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:58.935029 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:58.935931 (+ 902us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:58.935932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:58.936338 (+ 406us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:58.936747 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:58.936750 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:58.938343 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:58.938344 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:58.939246 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:58.939249 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:58.939954 (+ 705us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:58.939955 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:58.940040 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:58.940712 (+ 672us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:58.940716 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:58.942308 (+ 1592us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:58.945278 (+ 2970us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:58.945279 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:58.945279 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:58.945477 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:58.945717 (+ 240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:58.945718 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:58.945719 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:58.945782 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:58.946200 (+ 418us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:58.946208 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:58.946396 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 15:02:58.946526 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 15:02:58.946879 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":171,"mutex_wait_us":283,"thread_start_us":61,"threads_started":1}
I20260926 15:02:58.947306 23886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:58.934779 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:41671 (local address 127.8.102.1:50081)
0926 15:02:58.934854 (+ 75us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:58.934869 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:02:58.934934 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:02:58.935823 (+ 889us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:02:58.935826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:02:58.936303 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:02:58.936575 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:58.936582 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:58.938320 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:58.938321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:58.939248 (+ 927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:02:58.939251 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:02:58.939893 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:02:58.939895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:02:58.940040 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:58.940667 (+ 627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:02:58.940677 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:02:58.942787 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:02:58.944873 (+ 2086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:58.944882 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:58.944883 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:58.945184 (+ 301us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:58.945488 (+ 304us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:02:58.945489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:02:58.945489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:02:58.945651 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:02:58.946199 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:02:58.946208 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:02:58.946396 (+ 188us) client_negotiation.cc:769] Sending connection context
0926 15:02:58.946526 (+ 130us) client_negotiation.cc:240] Negotiation successful
0926 15:02:58.946818 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54,"mutex_wait_us":64}
I20260926 15:02:58.947367 23890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:58.934901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:50081 (local address 127.8.102.3:41671)
0926 15:02:58.935077 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0926 15:02:58.935087 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:02:58.935109 (+ 22us) server_negotiation.cc:407] Connection header received
0926 15:02:58.935541 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:02:58.935544 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:02:58.935597 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:02:58.935797 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:02:58.936738 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:58.938192 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:58.939367 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:02:58.939791 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:02:58.939881 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:02:58.942909 (+ 3028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:02:58.942913 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:02:58.942913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:02:58.942924 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:02:58.944776 (+ 1852us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:58.945288 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:58.945291 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:58.945291 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:58.945383 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:02:58.945752 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:02:58.945753 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:02:58.945753 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:02:58.946092 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:02:58.946185 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 15:02:58.947075 (+ 890us) server_negotiation.cc:299] Negotiation successful
0926 15:02:58.947241 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
W20260926 15:02:58.948066 23505 leader_election.cc:341] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43ca9df1eb32403fad905322acf736ca (127.8.102.2:36209): Not found: Tablet not found: 1a5ce8df704143c287f153891cdde3ad
W20260926 15:02:58.948321 23504 leader_election.cc:341] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0eabcd36a11845d5b9bc40918437fa3b (127.8.102.3:41671): Not found: Tablet not found: 1a5ce8df704143c287f153891cdde3ad
I20260926 15:02:58.948364 23504 leader_election.cc:302] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a452967a822457f83ef29ec496c8f60; no voters: 0eabcd36a11845d5b9bc40918437fa3b, 43ca9df1eb32403fad905322acf736ca
I20260926 15:02:58.948505 23888 raft_consensus.cc:2726] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:02:59.038869 23892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:59.038498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.3:51487)
0926 15:02:59.038761 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:59.038819 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":242,"thread_start_us":92,"threads_started":1}
I20260926 15:02:59.844612 23893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:59.844383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.1:56193)
0926 15:02:59.844521 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:59.844562 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":66,"threads_started":1}
I20260926 15:02:59.890339 23894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:02:59.890065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.2:55383)
0926 15:02:59.890238 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 15:02:59.890293 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153,"thread_start_us":97,"threads_started":1}
I20260926 15:03:00.039845 23895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:00.039544 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.3:58377)
0926 15:03:00.039749 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:00.039798 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":182,"thread_start_us":94,"threads_started":1}
I20260926 15:03:00.845378 23896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:00.845143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.1:34007)
0926 15:03:00.845283 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:00.845320 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":63,"threads_started":1}
I20260926 15:03:00.891248 23897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:00.890971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.2:58695)
0926 15:03:00.891128 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:00.891186 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":66,"threads_started":1}
I20260926 15:03:01.040669 23898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:01.040397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.3:51261)
0926 15:03:01.040557 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:01.040620 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260926 15:03:01.159859 23899 raft_consensus.cc:482] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:03:01.159961 23899 raft_consensus.cc:504] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } }
I20260926 15:03:01.160228 23899 leader_election.cc:288] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0eabcd36a11845d5b9bc40918437fa3b (127.8.102.3:41671), 43ca9df1eb32403fad905322acf736ca (127.8.102.2:36209)
W20260926 15:03:01.160701 23504 leader_election.cc:341] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0eabcd36a11845d5b9bc40918437fa3b (127.8.102.3:41671): Not found: Tablet not found: 1a5ce8df704143c287f153891cdde3ad
W20260926 15:03:01.160780 23505 leader_election.cc:341] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43ca9df1eb32403fad905322acf736ca (127.8.102.2:36209): Not found: Tablet not found: 1a5ce8df704143c287f153891cdde3ad
I20260926 15:03:01.160821 23505 leader_election.cc:302] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a452967a822457f83ef29ec496c8f60; no voters: 0eabcd36a11845d5b9bc40918437fa3b, 43ca9df1eb32403fad905322acf736ca
I20260926 15:03:01.160934 23899 raft_consensus.cc:2726] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:03:01.846207 23900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:01.845879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.1:58955)
0926 15:03:01.846108 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:01.846154 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209,"thread_start_us":78,"threads_started":1}
I20260926 15:03:01.892176 23901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:01.891938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.2:60703)
0926 15:03:01.892069 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:01.892116 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":66,"threads_started":1}
I20260926 15:03:02.041528 23902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:02.041283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.3:54613)
0926 15:03:02.041433 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:02.041479 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260926 15:03:02.847072 23903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:02.846863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.1:49379)
0926 15:03:02.846986 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:02.847025 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260926 15:03:02.893049 23904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:02.892772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.2:33211)
0926 15:03:02.892953 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:02.892998 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":79,"threads_started":1}
I20260926 15:03:03.042407 23905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:03.042129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.3:56497)
0926 15:03:03.042310 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:03.042357 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":83,"threads_started":1}
I20260926 15:03:03.176259 23906 raft_consensus.cc:482] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:03:03.176357 23906 raft_consensus.cc:504] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } }
I20260926 15:03:03.176601 23906 leader_election.cc:288] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0eabcd36a11845d5b9bc40918437fa3b (127.8.102.3:41671), 43ca9df1eb32403fad905322acf736ca (127.8.102.2:36209)
W20260926 15:03:03.177145 23504 leader_election.cc:341] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0eabcd36a11845d5b9bc40918437fa3b (127.8.102.3:41671): Not found: Tablet not found: 1a5ce8df704143c287f153891cdde3ad
W20260926 15:03:03.177256 23505 leader_election.cc:341] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43ca9df1eb32403fad905322acf736ca (127.8.102.2:36209): Not found: Tablet not found: 1a5ce8df704143c287f153891cdde3ad
I20260926 15:03:03.177332 23505 leader_election.cc:302] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a452967a822457f83ef29ec496c8f60; no voters: 0eabcd36a11845d5b9bc40918437fa3b, 43ca9df1eb32403fad905322acf736ca
I20260926 15:03:03.177405 23906 raft_consensus.cc:2726] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:03:03.847921 23907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:03.847645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.1:54777)
0926 15:03:03.847818 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:03.847862 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":67,"threads_started":1}
I20260926 15:03:03.893989 23908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:03.893715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.2:50339)
0926 15:03:03.893887 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:03.893935 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":72,"threads_started":1}
I20260926 15:03:04.043391 23909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:04.043088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.3:38251)
0926 15:03:04.043281 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:04.043332 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":93,"threads_started":1}
I20260926 15:03:04.848977 23910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:04.848694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.1:57903)
0926 15:03:04.848881 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:04.848927 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":88,"threads_started":1}
I20260926 15:03:04.894865 23911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:04.894570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.2:47259)
0926 15:03:04.894763 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:04.894817 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170,"thread_start_us":111,"threads_started":1}
I20260926 15:03:05.041718 23912 raft_consensus.cc:482] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:03:05.041858 23912 raft_consensus.cc:504] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } }
I20260926 15:03:05.042066 23912 leader_election.cc:288] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0eabcd36a11845d5b9bc40918437fa3b (127.8.102.3:41671), 43ca9df1eb32403fad905322acf736ca (127.8.102.2:36209)
W20260926 15:03:05.042526 23504 leader_election.cc:341] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0eabcd36a11845d5b9bc40918437fa3b (127.8.102.3:41671): Not found: Tablet not found: 1a5ce8df704143c287f153891cdde3ad
W20260926 15:03:05.042610 23505 leader_election.cc:341] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43ca9df1eb32403fad905322acf736ca (127.8.102.2:36209): Not found: Tablet not found: 1a5ce8df704143c287f153891cdde3ad
I20260926 15:03:05.042716 23505 leader_election.cc:302] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a452967a822457f83ef29ec496c8f60; no voters: 0eabcd36a11845d5b9bc40918437fa3b, 43ca9df1eb32403fad905322acf736ca
I20260926 15:03:05.042795 23912 raft_consensus.cc:2726] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:03:05.044135 23913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:05.043909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.3:55675)
0926 15:03:05.044054 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:05.044094 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":82,"threads_started":1}
I20260926 15:03:05.849865 23914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:05.849642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.1:39173)
0926 15:03:05.849771 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:05.849815 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260926 15:03:05.895812 23915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:05.895521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.2:44567)
0926 15:03:05.895723 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:05.895763 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":174,"thread_start_us":137,"threads_started":1}
I20260926 15:03:06.044904 23916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:06.044626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.3:60075)
0926 15:03:06.044780 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:06.044829 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":62,"threads_started":1}
I20260926 15:03:06.850802 23917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:06.850530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.1:39825)
0926 15:03:06.850696 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:06.850752 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":83,"threads_started":1}
I20260926 15:03:06.896752 23918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:06.896499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.2:41585)
0926 15:03:06.896656 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:06.896704 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260926 15:03:07.045809 23919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:07.045511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.3:49141)
0926 15:03:07.045714 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:07.045762 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":180,"thread_start_us":106,"threads_started":1}
I20260926 15:03:07.851811 23920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:07.851540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.1:33227)
0926 15:03:07.851711 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:07.851763 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":68,"threads_started":1}
I20260926 15:03:07.897684 23921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:07.897400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.2:49591)
0926 15:03:07.897583 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:07.897633 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163,"thread_start_us":75,"threads_started":1}
I20260926 15:03:08.046813 23922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:08.046503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.3:33465)
0926 15:03:08.046710 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:08.046761 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":78,"threads_started":1}
I20260926 15:03:08.852717 23923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:08.852381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.1:41313)
0926 15:03:08.852564 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:08.852634 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":74,"threads_started":1}
I20260926 15:03:08.898594 23924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:08.898303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.2:40221)
0926 15:03:08.898490 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:08.898539 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164,"thread_start_us":78,"threads_started":1}
I20260926 15:03:09.047806 23925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:09.047525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.3:38981)
0926 15:03:09.047704 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:09.047752 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":69,"threads_started":1}
I20260926 15:03:09.853583 23926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:09.853299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.1:47131)
0926 15:03:09.853487 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:09.853533 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165,"thread_start_us":81,"threads_started":1}
I20260926 15:03:09.899519 23927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:09.899246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.2:38671)
0926 15:03:09.899423 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:09.899470 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":156,"thread_start_us":78,"threads_started":1}
I20260926 15:03:10.048740 23928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:10.048471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.3:37253)
0926 15:03:10.048646 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:10.048692 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":91,"threads_started":1}
I20260926 15:03:10.854383 23929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:10.854143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.1:43297)
0926 15:03:10.854303 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:10.854336 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":99,"threads_started":1}
I20260926 15:03:10.900403 23930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:10.900169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.2:32951)
0926 15:03:10.900312 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:10.900354 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":66,"threads_started":1}
I20260926 15:03:11.049580 23931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:11.049318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.3:37999)
0926 15:03:11.049489 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:11.049533 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":83,"threads_started":1}
I20260926 15:03:11.732446 23932 raft_consensus.cc:482] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:03:11.732654 23932 raft_consensus.cc:504] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0eabcd36a11845d5b9bc40918437fa3b" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 41671 } } peers { permanent_uuid: "43ca9df1eb32403fad905322acf736ca" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 36209 } } peers { permanent_uuid: "7a452967a822457f83ef29ec496c8f60" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 34853 } }
I20260926 15:03:11.732955 23932 leader_election.cc:288] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 0eabcd36a11845d5b9bc40918437fa3b (127.8.102.3:41671), 43ca9df1eb32403fad905322acf736ca (127.8.102.2:36209)
W20260926 15:03:11.733445 23504 leader_election.cc:341] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 0eabcd36a11845d5b9bc40918437fa3b (127.8.102.3:41671): Not found: Tablet not found: 1a5ce8df704143c287f153891cdde3ad
W20260926 15:03:11.733520 23505 leader_election.cc:341] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 43ca9df1eb32403fad905322acf736ca (127.8.102.2:36209): Not found: Tablet not found: 1a5ce8df704143c287f153891cdde3ad
I20260926 15:03:11.733592 23505 leader_election.cc:302] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7a452967a822457f83ef29ec496c8f60; no voters: 0eabcd36a11845d5b9bc40918437fa3b, 43ca9df1eb32403fad905322acf736ca
I20260926 15:03:11.733655 23932 raft_consensus.cc:2726] T 1a5ce8df704143c287f153891cdde3ad P 7a452967a822457f83ef29ec496c8f60 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:03:11.855156 23933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:11.854929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.1:53559)
0926 15:03:11.855065 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:11.855109 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":75,"threads_started":1}
I20260926 15:03:11.901196 23934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:11.900949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.2:50231)
0926 15:03:11.901105 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:11.901150 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":80,"threads_started":1}
I20260926 15:03:12.050467 23935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:12.050219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.3:40985)
0926 15:03:12.050369 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:12.050417 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260926 15:03:12.855994 23936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:12.855748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.1:52109)
0926 15:03:12.855895 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:12.855936 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20260926 15:03:12.902014 23937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:12.901782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.2:52341)
0926 15:03:12.901922 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:12.901967 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260926 15:03:13.043110 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 23457
I20260926 15:03:13.047767 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 23621
I20260926 15:03:13.051404 23938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:13.051196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:45439 (local address 127.8.102.3:34571)
0926 15:03:13.051329 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:13.051362 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:45439: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260926 15:03:13.052209 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 23753
2026-09-26T15:03:13Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (18610 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:03:13 dist-test-slave-jnhs krb5kdc[23941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:03:13 dist-test-slave-jnhs krb5kdc[23941](info): set up 2 sockets
Sep 26 15:03:13 dist-test-slave-jnhs krb5kdc[23941](info): commencing operation
krb5kdc: starting...
W20260926 15:03:14.130911 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.058s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:03:14 dist-test-slave-jnhs krb5kdc[23941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:03:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:03:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:03:14.257629 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:36407
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:36407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:03:14.341547 23954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:03:14.341770 23954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:03:14.341794 23954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:03:14.343083 23954 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 15:03:14.343132 23954 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 15:03:14.343144 23954 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 15:03:14.343156 23954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:03:14.343171 23954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:03:14.343185 23954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:03:14.343197 23954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:03:14.343209 23954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:03:14.344698 23954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:36407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:36407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:03:14.344910 23954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:03:14.345094 23954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:03:14.350488 23960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:03:14.350523 23959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:03:14.350666 23954 server_base.cc:1034] running on GCE node
W20260926 15:03:14.350452 23962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:03:14.351073 23954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:03:14.351389 23954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:03:14.352628 23954 hybrid_clock.cc:648] HybridClock initialized: now 1790434994352615 us; error 39 us; skew 500 ppm
Sep 26 15:03:14 dist-test-slave-jnhs krb5kdc[23941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434994, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:03:14.356827 23954 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:03:14.357333 23954 webserver.cc:460] Webserver started at http://127.8.102.62:34717/ using document root <none> and password file <none>
I20260926 15:03:14.357571 23954 fs_manager.cc:359] Metadata directory not provided
I20260926 15:03:14.357642 23954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:03:14.357796 23954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:03:14.358980 23954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "14791678bc4b4a908632f5923634feff"
format_stamp: "Formatted at 2026-09-26 15:03:14 on dist-test-slave-jnhs"
server_key: "767991b628f7f2eced21d862f35c87d8"
server_key_iv: "e2f6abe6b4a4d1ba5702433af036abcf"
server_key_version: "encryptionkey@0"
I20260926 15:03:14.359378 23954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "14791678bc4b4a908632f5923634feff"
format_stamp: "Formatted at 2026-09-26 15:03:14 on dist-test-slave-jnhs"
server_key: "767991b628f7f2eced21d862f35c87d8"
server_key_iv: "e2f6abe6b4a4d1ba5702433af036abcf"
server_key_version: "encryptionkey@0"
I20260926 15:03:14.360816 23954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:03:14.361815 23969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:14.362097 23954 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:03:14.362237 23954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "14791678bc4b4a908632f5923634feff"
format_stamp: "Formatted at 2026-09-26 15:03:14 on dist-test-slave-jnhs"
server_key: "767991b628f7f2eced21d862f35c87d8"
server_key_iv: "e2f6abe6b4a4d1ba5702433af036abcf"
server_key_version: "encryptionkey@0"
I20260926 15:03:14.362355 23954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:03:14.383512 23954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:03:14.386021 23954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:03:14.386152 23954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:03:14.390566 23954 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:36407
I20260926 15:03:14.390619 24021 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:36407 every 8 connection(s)
I20260926 15:03:14.391036 23954 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:03:14.391999 24022 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:03:14.393026 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 23954
I20260926 15:03:14.393154 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:03:14.393347 8600 external_mini_cluster.cc:1428] Setting key 5c53bb9c02ddd8c6c70bf248d976adf2
I20260926 15:03:14.394966 24022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff: Bootstrap starting.
I20260926 15:03:14.395761 24022 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff: Neither blocks nor log segments found. Creating new log.
I20260926 15:03:14.396088 24022 log.cc:824] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff: Log is configured to *not* fsync() on all Append() calls
I20260926 15:03:14.396790 24022 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff: No bootstrap required, opened a new log
I20260926 15:03:14.398824 24022 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14791678bc4b4a908632f5923634feff" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36407 } }
I20260926 15:03:14.398998 24022 raft_consensus.cc:374] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:03:14.399050 24022 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14791678bc4b4a908632f5923634feff, State: Initialized, Role: FOLLOWER
I20260926 15:03:14.399178 24022 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14791678bc4b4a908632f5923634feff" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36407 } }
I20260926 15:03:14.399274 24022 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:03:14.399319 24022 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:03:14.399365 24022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:03:14.399961 24022 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14791678bc4b4a908632f5923634feff" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36407 } }
I20260926 15:03:14.400079 24022 leader_election.cc:302] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14791678bc4b4a908632f5923634feff; no voters:
I20260926 15:03:14.400234 24022 leader_election.cc:288] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:03:14.400261 24027 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:03:14.400470 24027 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 1 LEADER]: Becoming Leader. State: Replica: 14791678bc4b4a908632f5923634feff, State: Running, Role: LEADER
Sep 26 15:03:14 dist-test-slave-jnhs krb5kdc[23941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434994, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:03:14.400532 24022 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:03:14.400588 24027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14791678bc4b4a908632f5923634feff" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36407 } }
I20260926 15:03:14.400985 24027 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14791678bc4b4a908632f5923634feff. Latest consensus state: current_term: 1 leader_uuid: "14791678bc4b4a908632f5923634feff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14791678bc4b4a908632f5923634feff" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36407 } } }
I20260926 15:03:14.400992 24028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "14791678bc4b4a908632f5923634feff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14791678bc4b4a908632f5923634feff" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36407 } } }
I20260926 15:03:14.401160 24028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [sys.catalog]: This master's current role is: LEADER
I20260926 15:03:14.401285 24027 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [sys.catalog]: This master's current role is: LEADER
I20260926 15:03:14.401710 24036 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:03:14.402271 24036 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:03:14.405213 24036 catalog_manager.cc:1347] Generated new cluster ID: 0accf826d80b411497c1d0b95062ad1a
I20260926 15:03:14.405270 24036 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:03:14.405877 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:14.393917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34382 (local address 127.8.102.62:36407)
0926 15:03:14.394110 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:14.394118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:14.394142 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:03:14.394712 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:14.394719 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:14.394771 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:14.394887 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:03:14.395773 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:14.396797 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:14.397618 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:14.398156 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:14.398280 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:14.401135 (+ 2855us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:03:14.401140 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:03:14.401140 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:03:14.401152 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:03:14.404069 (+ 2917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:14.404456 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:14.404460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:14.404460 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:14.404574 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:14.404829 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:14.404830 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:14.404830 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:14.405075 (+ 245us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:03:14.405230 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:14.405495 (+ 265us) server_negotiation.cc:299] Negotiation successful
0926 15:03:14.405642 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":110,"threads_started":1}
I20260926 15:03:14.449658 24036 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:03:14.450512 24036 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:03:14.456382 24036 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff: Generated new TSK 0
I20260926 15:03:14.456602 24036 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
I20260926 15:03:14.494597 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36407
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:03:14.596309 24048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:03:14.596513 24048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:03:14.596537 24048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:03:14.597725 24048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:03:14.597775 24048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:03:14.597803 24048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:03:14.599362 24048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24048
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:03:14.599620 24048 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:03:14.599923 24048 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:03:14.605144 24056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:03:14.605144 24054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:03:14.605203 24053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:03:14.605453 24048 server_base.cc:1034] running on GCE node
I20260926 15:03:14.605643 24048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:03:14.605871 24048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:03:14.607066 24048 hybrid_clock.cc:648] HybridClock initialized: now 1790434994607025 us; error 39 us; skew 500 ppm
Sep 26 15:03:14 dist-test-slave-jnhs krb5kdc[23941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434994, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:03:14.609807 24048 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:03:14.610349 24048 webserver.cc:460] Webserver started at http://127.8.102.1:34731/ using document root <none> and password file <none>
I20260926 15:03:14.610582 24048 fs_manager.cc:359] Metadata directory not provided
I20260926 15:03:14.610684 24048 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:03:14.610845 24048 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:03:14.611999 24048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "9cf31f40b5b34e36b5840dead2e75f4f"
format_stamp: "Formatted at 2026-09-26 15:03:14 on dist-test-slave-jnhs"
server_key: "99643f6f2998539a8e74a1b6aee9e8fd"
server_key_iv: "6e375f1638b80e145d4b328e5359d180"
server_key_version: "encryptionkey@0"
I20260926 15:03:14.612443 24048 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "9cf31f40b5b34e36b5840dead2e75f4f"
format_stamp: "Formatted at 2026-09-26 15:03:14 on dist-test-slave-jnhs"
server_key: "99643f6f2998539a8e74a1b6aee9e8fd"
server_key_iv: "6e375f1638b80e145d4b328e5359d180"
server_key_version: "encryptionkey@0"
I20260926 15:03:14.613912 24048 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 15:03:14.614806 24063 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:14.615051 24048 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:03:14.615157 24048 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "9cf31f40b5b34e36b5840dead2e75f4f"
format_stamp: "Formatted at 2026-09-26 15:03:14 on dist-test-slave-jnhs"
server_key: "99643f6f2998539a8e74a1b6aee9e8fd"
server_key_iv: "6e375f1638b80e145d4b328e5359d180"
server_key_version: "encryptionkey@0"
I20260926 15:03:14.615235 24048 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:03:14.633394 24048 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:03:14.636916 24048 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:03:14.637135 24048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:03:14.637418 24048 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:03:14.637810 24048 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:03:14.637867 24048 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:14.637905 24048 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:03:14.637938 24048 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:14.644219 24048 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:43347
I20260926 15:03:14.644315 24176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:43347 every 8 connection(s)
I20260926 15:03:14.644645 24048 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:03:14.650710 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 24048
I20260926 15:03:14.650846 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:03:14.651043 8600 external_mini_cluster.cc:1428] Setting key b34e154503b279b0a45e8b9c84c3c2d7
Sep 26 15:03:14 dist-test-slave-jnhs krb5kdc[23941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434994, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:03:14.654824 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:14.645442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:50025 (local address 127.8.102.62:36407)
0926 15:03:14.645536 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:14.645538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:14.645727 (+ 189us) server_negotiation.cc:407] Connection header received
0926 15:03:14.646330 (+ 603us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:14.646331 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:14.646345 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:14.646414 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:03:14.647469 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:14.647989 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:14.649024 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:14.649246 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:14.649326 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:14.651856 (+ 2530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:03:14.651857 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:03:14.651858 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:03:14.651864 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:03:14.653224 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:14.653773 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:14.653774 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:14.653775 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:14.653835 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:14.654090 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:14.654091 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:14.654092 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:14.654217 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:03:14.654296 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:14.654618 (+ 322us) server_negotiation.cc:299] Negotiation successful
0926 15:03:14.654760 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260926 15:03:14.655078 24179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:14.645392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.1:50025)
0926 15:03:14.645593 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:14.645644 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 15:03:14.646205 (+ 561us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:03:14.646470 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:03:14.646474 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:03:14.647157 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:03:14.647358 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:14.647364 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:14.648092 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:14.648093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:14.648918 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:14.648920 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:14.649331 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:14.649333 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:14.649472 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:14.649977 (+ 505us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:03:14.649986 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:03:14.651652 (+ 1666us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:03:14.653434 (+ 1782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:03:14.653441 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:03:14.653443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:03:14.653679 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:03:14.653904 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:03:14.653905 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:03:14.653906 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:03:14.654019 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:03:14.654287 (+ 268us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:03:14.654294 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:03:14.654463 (+ 169us) client_negotiation.cc:769] Sending connection context
0926 15:03:14.654577 (+ 114us) client_negotiation.cc:240] Negotiation successful
0926 15:03:14.654797 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":83,"threads_started":1}
I20260926 15:03:14.655591 24177 heartbeater.cc:347] Connected to a master server at 127.8.102.62:36407
I20260926 15:03:14.655689 24177 heartbeater.cc:464] Registering TS with master...
I20260926 15:03:14.655889 24177 heartbeater.cc:511] Master 127.8.102.62:36407 requested a full tablet report, sending...
I20260926 15:03:14.656445 23986 ts_manager.cc:194] Registered new tserver with Master: 9cf31f40b5b34e36b5840dead2e75f4f (127.8.102.1:43347)
I20260926 15:03:14.657048 23986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:50025
WARNING: no policy specified for kudu/127.8.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
I20260926 15:03:14.677716 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36407
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:03:14.765273 24182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:03:14.765497 24182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:03:14.765524 24182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:03:14.766782 24182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:03:14.766827 24182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:03:14.766855 24182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:03:14.768456 24182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24182
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:03:14.768715 24182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:03:14.768961 24182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:03:14.774201 24188 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:03:14.774175 24190 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:03:14.774197 24187 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:03:14.774606 24182 server_base.cc:1034] running on GCE node
I20260926 15:03:14.774857 24182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:03:14.775123 24182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:03:14.776285 24182 hybrid_clock.cc:648] HybridClock initialized: now 1790434994776272 us; error 33 us; skew 500 ppm
Sep 26 15:03:14 dist-test-slave-jnhs krb5kdc[23941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434994, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:03:14.778947 24182 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:03:14.779605 24182 webserver.cc:460] Webserver started at http://127.8.102.2:38937/ using document root <none> and password file <none>
I20260926 15:03:14.779860 24182 fs_manager.cc:359] Metadata directory not provided
I20260926 15:03:14.779937 24182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:03:14.780115 24182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:03:14.781322 24182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "b339edec4d20486393117d522b04f410"
format_stamp: "Formatted at 2026-09-26 15:03:14 on dist-test-slave-jnhs"
server_key: "68227588a6454cddf2400aa727e9a942"
server_key_iv: "fb7a6b9f88c55b4d547355cf7acc4e2b"
server_key_version: "encryptionkey@0"
I20260926 15:03:14.781735 24182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "b339edec4d20486393117d522b04f410"
format_stamp: "Formatted at 2026-09-26 15:03:14 on dist-test-slave-jnhs"
server_key: "68227588a6454cddf2400aa727e9a942"
server_key_iv: "fb7a6b9f88c55b4d547355cf7acc4e2b"
server_key_version: "encryptionkey@0"
I20260926 15:03:14.783419 24182 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 15:03:14.784315 24197 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:14.784513 24182 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:03:14.784622 24182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "b339edec4d20486393117d522b04f410"
format_stamp: "Formatted at 2026-09-26 15:03:14 on dist-test-slave-jnhs"
server_key: "68227588a6454cddf2400aa727e9a942"
server_key_iv: "fb7a6b9f88c55b4d547355cf7acc4e2b"
server_key_version: "encryptionkey@0"
I20260926 15:03:14.784710 24182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:03:14.807336 24182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:03:14.809541 24182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:03:14.809759 24182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:03:14.810087 24182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:03:14.810544 24182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:03:14.810611 24182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:14.810686 24182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:03:14.810711 24182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:14.816715 24182 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:33963
I20260926 15:03:14.816802 24310 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:33963 every 8 connection(s)
I20260926 15:03:14.817140 24182 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:03:14.823203 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 24182
I20260926 15:03:14.823341 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:03:14.823580 8600 external_mini_cluster.cc:1428] Setting key 42085fa28c6f66f7d86a208d0dc38368
Sep 26 15:03:14 dist-test-slave-jnhs krb5kdc[23941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434994, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:03:14.828009 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:14.817909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:33193 (local address 127.8.102.62:36407)
0926 15:03:14.818004 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:14.818007 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:14.818223 (+ 216us) server_negotiation.cc:407] Connection header received
0926 15:03:14.818845 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:14.818846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:14.818857 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:14.818975 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:03:14.820022 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:14.820597 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:14.821632 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:14.821844 (+ 212us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:14.822117 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:14.824620 (+ 2503us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:03:14.824621 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:03:14.824622 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:03:14.824631 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:03:14.826292 (+ 1661us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:14.826860 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:14.826861 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:14.826861 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:14.826916 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:14.827230 (+ 314us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:14.827231 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:14.827231 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:14.827378 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:03:14.827444 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:14.827770 (+ 326us) server_negotiation.cc:299] Negotiation successful
0926 15:03:14.827867 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 15:03:14.828029 24313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:14.817902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.2:33193)
0926 15:03:14.818088 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:14.818125 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 15:03:14.818743 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:03:14.819072 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:03:14.819076 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:03:14.819723 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:03:14.819920 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:14.819924 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:14.820692 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:14.820692 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:14.821533 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:14.821536 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:14.821946 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:14.821947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:14.822090 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:14.822616 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:03:14.822625 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:03:14.824508 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:03:14.826377 (+ 1869us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:03:14.826385 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:03:14.826387 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:03:14.826784 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:03:14.826993 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:03:14.826994 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:03:14.826995 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:03:14.827144 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:03:14.827465 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:03:14.827472 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:03:14.827619 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 15:03:14.827710 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 15:03:14.827843 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147,"thread_start_us":92,"threads_started":1}
I20260926 15:03:14.828579 24311 heartbeater.cc:347] Connected to a master server at 127.8.102.62:36407
I20260926 15:03:14.828719 24311 heartbeater.cc:464] Registering TS with master...
I20260926 15:03:14.828938 24311 heartbeater.cc:511] Master 127.8.102.62:36407 requested a full tablet report, sending...
I20260926 15:03:14.829334 23986 ts_manager.cc:194] Registered new tserver with Master: b339edec4d20486393117d522b04f410 (127.8.102.2:33963)
I20260926 15:03:14.829730 23986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:33193
WARNING: no policy specified for kudu/127.8.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
I20260926 15:03:14.854522 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36407
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:03:14.941892 24316 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:03:14.942121 24316 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:03:14.942144 24316 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:03:14.943442 24316 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:03:14.943492 24316 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:03:14.943521 24316 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:03:14.945066 24316 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:36407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24316
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:03:14.945325 24316 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:03:14.945585 24316 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:03:14.951020 24322 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:03:14.950991 24321 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:03:14.950991 24324 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:03:14.951134 24316 server_base.cc:1034] running on GCE node
I20260926 15:03:14.951484 24316 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:03:14.951725 24316 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:03:14.952910 24316 hybrid_clock.cc:648] HybridClock initialized: now 1790434994952890 us; error 27 us; skew 500 ppm
Sep 26 15:03:14 dist-test-slave-jnhs krb5kdc[23941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434994, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:03:14.956163 24316 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:03:14.956710 24316 webserver.cc:460] Webserver started at http://127.8.102.3:38469/ using document root <none> and password file <none>
I20260926 15:03:14.956976 24316 fs_manager.cc:359] Metadata directory not provided
I20260926 15:03:14.957047 24316 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:03:14.957201 24316 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:03:14.958551 24316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "1ef46f3beb7049288f0dd6d2de848a71"
format_stamp: "Formatted at 2026-09-26 15:03:14 on dist-test-slave-jnhs"
server_key: "310ac9bb844cc46ad996f211e24fd6f4"
server_key_iv: "1f945a0adee040a3bb88b30e6b70fea5"
server_key_version: "encryptionkey@0"
I20260926 15:03:14.958953 24316 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "1ef46f3beb7049288f0dd6d2de848a71"
format_stamp: "Formatted at 2026-09-26 15:03:14 on dist-test-slave-jnhs"
server_key: "310ac9bb844cc46ad996f211e24fd6f4"
server_key_iv: "1f945a0adee040a3bb88b30e6b70fea5"
server_key_version: "encryptionkey@0"
I20260926 15:03:14.960332 24316 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:03:14.961025 24331 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:14.961215 24316 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 15:03:14.961308 24316 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "1ef46f3beb7049288f0dd6d2de848a71"
format_stamp: "Formatted at 2026-09-26 15:03:14 on dist-test-slave-jnhs"
server_key: "310ac9bb844cc46ad996f211e24fd6f4"
server_key_iv: "1f945a0adee040a3bb88b30e6b70fea5"
server_key_version: "encryptionkey@0"
I20260926 15:03:14.961383 24316 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:03:14.984787 24316 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:03:14.986843 24316 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:03:14.987010 24316 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:03:14.987269 24316 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:03:14.987746 24316 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:03:14.987785 24316 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:14.987809 24316 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:03:14.987821 24316 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:14.993999 24316 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:46003
I20260926 15:03:14.994042 24444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:46003 every 8 connection(s)
I20260926 15:03:14.994382 24316 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:03:15.000165 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 24316
I20260926 15:03:15.000290 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:03:15.000500 8600 external_mini_cluster.cc:1428] Setting key 1b20e391ae66ee40f3bcd83bc865fcde
Sep 26 15:03:14 dist-test-slave-jnhs krb5kdc[23941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790434994, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:03:15.004639 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:14.995133 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:50625 (local address 127.8.102.62:36407)
0926 15:03:14.995204 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:14.995206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:14.995417 (+ 211us) server_negotiation.cc:407] Connection header received
0926 15:03:14.995910 (+ 493us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:14.995911 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:14.995920 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:14.995992 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:03:14.996992 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:14.997505 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:14.998511 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:14.998736 (+ 225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:14.998797 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:15.001508 (+ 2711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:03:15.001509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:03:15.001510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:03:15.001518 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:03:15.003083 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:15.003551 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:15.003552 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:15.003553 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:15.003614 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:15.003913 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:15.003914 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:15.003915 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:15.004042 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:03:15.004133 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:15.004496 (+ 363us) server_negotiation.cc:299] Negotiation successful
0926 15:03:15.004571 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 15:03:15.004782 24447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:14.995093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.3:50625)
0926 15:03:14.995288 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:14.995331 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:03:14.995827 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:03:14.996106 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:03:14.996110 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:03:14.996690 (+ 580us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:03:14.996887 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:14.996891 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:14.997599 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:14.997599 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:14.998417 (+ 818us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:14.998419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:14.998819 (+ 400us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:14.998820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:14.998935 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:14.999498 (+ 563us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:03:14.999509 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:03:15.001353 (+ 1844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:03:15.003182 (+ 1829us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:03:15.003189 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:03:15.003191 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:03:15.003446 (+ 255us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:03:15.003708 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:03:15.003709 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:03:15.003710 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:03:15.003824 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:03:15.004138 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:03:15.004146 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:03:15.004295 (+ 149us) client_negotiation.cc:769] Sending connection context
0926 15:03:15.004394 (+ 99us) client_negotiation.cc:240] Negotiation successful
0926 15:03:15.004536 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":119,"threads_started":1}
I20260926 15:03:15.005278 24445 heartbeater.cc:347] Connected to a master server at 127.8.102.62:36407
I20260926 15:03:15.005400 24445 heartbeater.cc:464] Registering TS with master...
I20260926 15:03:15.005589 24445 heartbeater.cc:511] Master 127.8.102.62:36407 requested a full tablet report, sending...
I20260926 15:03:15.006021 23986 ts_manager.cc:194] Registered new tserver with Master: 1ef46f3beb7049288f0dd6d2de848a71 (127.8.102.3:46003)
I20260926 15:03:15.006366 23986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:50625
I20260926 15:03:15.013010 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:03:15.021569 24025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:15.014544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34398 (local address 127.8.102.62:36407)
0926 15:03:15.014705 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:15.014708 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:15.014822 (+ 114us) server_negotiation.cc:407] Connection header received
0926 15:03:15.014853 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:15.014854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:15.014865 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:15.014929 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:03:15.015596 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.016133 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.017053 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.017345 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.017426 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:15.018887 (+ 1461us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:03:15.018888 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:03:15.018889 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:03:15.018897 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:03:15.020355 (+ 1458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:15.020695 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:15.020695 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:15.020696 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:15.020749 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:15.020976 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:15.020977 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:15.020977 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:15.021158 (+ 181us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:03:15.021265 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:15.021407 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 15:03:15.021477 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:03:15.023712 23986 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34398:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:03:15.025362 23986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:03:15.033588 24456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:15.028518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:43347 (local address 127.0.0.1:42344)
0926 15:03:15.028921 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:15.028965 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:03:15.029231 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:03:15.029426 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:03:15.029429 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:03:15.029432 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:03:15.029511 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:15.029513 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.030529 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.030530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:15.032226 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:15.032230 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.033242 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.033244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:15.033338 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:15.033354 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 15:03:15.033432 (+ 78us) client_negotiation.cc:240] Negotiation successful
0926 15:03:15.033491 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":393,"mutex_wait_us":28,"thread_start_us":100,"threads_started":1}
I20260926 15:03:15.033872 24458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:15.028693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42344 (local address 127.8.102.1:43347)
0926 15:03:15.028906 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:15.028911 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:15.029192 (+ 281us) server_negotiation.cc:407] Connection header received
0926 15:03:15.029321 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:15.029324 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:15.029341 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:15.029415 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:03:15.029615 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.030421 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.032371 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.033139 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.033219 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:15.033281 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:15.033734 (+ 453us) server_negotiation.cc:299] Negotiation successful
0926 15:03:15.033784 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":97,"threads_started":1}
I20260926 15:03:15.034315 24457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:15.028518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:46003 (local address 127.0.0.1:58844)
0926 15:03:15.028920 (+ 402us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:15.028965 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 15:03:15.029037 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:03:15.029616 (+ 579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:03:15.029617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:03:15.029618 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:03:15.029670 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:15.029672 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.031060 (+ 1388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.031061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:15.032264 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:15.032266 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.034037 (+ 1771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.034038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:15.034112 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:15.034117 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:03:15.034186 (+ 69us) client_negotiation.cc:240] Negotiation successful
0926 15:03:15.034231 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":368,"spinlock_wait_cycles":768,"thread_start_us":108,"threads_started":1}
I20260926 15:03:15.034854 24460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:15.028743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58844 (local address 127.8.102.3:46003)
0926 15:03:15.029194 (+ 451us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:15.029198 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:15.029207 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:03:15.029258 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:15.029260 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:15.029278 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:15.029358 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:03:15.029752 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.030931 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.032377 (+ 1446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.033136 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.033219 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:15.033281 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:15.034725 (+ 1444us) server_negotiation.cc:299] Negotiation successful
0926 15:03:15.034776 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":100,"threads_started":1}
I20260926 15:03:15.034716 24111 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5096d6131649f3b8ff0568d5cffd3f (DEFAULT_TABLE table=test-table [id=2434c9bbf5db4687b46a1e5e618b0a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:03:15.035043 24461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:15.028906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58634 (local address 127.8.102.2:33963)
0926 15:03:15.029088 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:15.029092 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:15.029550 (+ 458us) server_negotiation.cc:407] Connection header received
0926 15:03:15.029619 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:15.029622 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:15.029636 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:15.029718 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:03:15.029827 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.030930 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.033302 (+ 2372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.034114 (+ 812us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.034194 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:15.034253 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:15.034917 (+ 664us) server_negotiation.cc:299] Negotiation successful
0926 15:03:15.034963 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":152,"thread_start_us":107,"threads_started":1}
I20260926 15:03:15.035054 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:15.028772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:33963 (local address 127.0.0.1:58634)
0926 15:03:15.029468 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:15.029477 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 15:03:15.029547 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:03:15.029728 (+ 181us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:03:15.029729 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:03:15.029731 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:03:15.029767 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:15.029769 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.032391 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.032391 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:15.033165 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:15.033167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.034736 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.034737 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:15.034809 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:15.034813 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:03:15.034896 (+ 83us) client_negotiation.cc:240] Negotiation successful
0926 15:03:15.034938 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":78,"threads_started":1}
I20260926 15:03:15.035115 24111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5096d6131649f3b8ff0568d5cffd3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:03:15.035801 24245 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5096d6131649f3b8ff0568d5cffd3f (DEFAULT_TABLE table=test-table [id=2434c9bbf5db4687b46a1e5e618b0a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:03:15.035952 24379 tablet_service.cc:1467] Processing CreateTablet for tablet 6c5096d6131649f3b8ff0568d5cffd3f (DEFAULT_TABLE table=test-table [id=2434c9bbf5db4687b46a1e5e618b0a2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:03:15.036185 24245 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5096d6131649f3b8ff0568d5cffd3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:03:15.036265 24379 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6c5096d6131649f3b8ff0568d5cffd3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:03:15.038163 24462 tablet_bootstrap.cc:492] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f: Bootstrap starting.
I20260926 15:03:15.038872 24462 tablet_bootstrap.cc:654] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f: Neither blocks nor log segments found. Creating new log.
I20260926 15:03:15.039386 24462 log.cc:824] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f: Log is configured to *not* fsync() on all Append() calls
I20260926 15:03:15.039664 24463 tablet_bootstrap.cc:492] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410: Bootstrap starting.
I20260926 15:03:15.039873 24464 tablet_bootstrap.cc:492] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71: Bootstrap starting.
I20260926 15:03:15.040527 24464 tablet_bootstrap.cc:654] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71: Neither blocks nor log segments found. Creating new log.
I20260926 15:03:15.040618 24462 tablet_bootstrap.cc:492] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f: No bootstrap required, opened a new log
I20260926 15:03:15.040704 24462 ts_tablet_manager.cc:1397] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 15:03:15.041059 24464 log.cc:824] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71: Log is configured to *not* fsync() on all Append() calls
I20260926 15:03:15.041827 24463 tablet_bootstrap.cc:654] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410: Neither blocks nor log segments found. Creating new log.
I20260926 15:03:15.041824 24464 tablet_bootstrap.cc:492] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71: No bootstrap required, opened a new log
I20260926 15:03:15.041919 24464 ts_tablet_manager.cc:1397] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:03:15.042330 24463 log.cc:824] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410: Log is configured to *not* fsync() on all Append() calls
I20260926 15:03:15.042704 24462 raft_consensus.cc:348] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:15.042907 24462 raft_consensus.cc:374] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:03:15.042963 24462 raft_consensus.cc:729] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf31f40b5b34e36b5840dead2e75f4f, State: Initialized, Role: FOLLOWER
I20260926 15:03:15.043085 24462 consensus_queue.cc:260] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:15.043226 24463 tablet_bootstrap.cc:492] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410: No bootstrap required, opened a new log
I20260926 15:03:15.043286 24463 ts_tablet_manager.cc:1397] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.000s
I20260926 15:03:15.043367 24462 ts_tablet_manager.cc:1428] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:03:15.043545 24177 heartbeater.cc:503] Master 127.8.102.62:36407 was elected leader, sending a full tablet report...
I20260926 15:03:15.044183 24464 raft_consensus.cc:348] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:15.044404 24464 raft_consensus.cc:374] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:03:15.044446 24464 raft_consensus.cc:729] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ef46f3beb7049288f0dd6d2de848a71, State: Initialized, Role: FOLLOWER
I20260926 15:03:15.044591 24464 consensus_queue.cc:260] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:15.044873 24464 ts_tablet_manager.cc:1428] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:03:15.044941 24445 heartbeater.cc:503] Master 127.8.102.62:36407 was elected leader, sending a full tablet report...
I20260926 15:03:15.045323 24463 raft_consensus.cc:348] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:15.045534 24463 raft_consensus.cc:374] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:03:15.045593 24463 raft_consensus.cc:729] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b339edec4d20486393117d522b04f410, State: Initialized, Role: FOLLOWER
I20260926 15:03:15.045712 24463 consensus_queue.cc:260] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:15.046020 24463 ts_tablet_manager.cc:1428] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:03:15.046159 24311 heartbeater.cc:503] Master 127.8.102.62:36407 was elected leader, sending a full tablet report...
W20260926 15:03:15.067709 24312 tablet.cc:2369] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:03:15.145401 24178 tablet.cc:2369] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:03:15.221596 24468 raft_consensus.cc:482] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:03:15.221788 24468 raft_consensus.cc:504] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:15.222235 24468 leader_election.cc:288] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1ef46f3beb7049288f0dd6d2de848a71 (127.8.102.3:46003), b339edec4d20486393117d522b04f410 (127.8.102.2:33963)
I20260926 15:03:15.226711 24461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:15.222926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:36057 (local address 127.8.102.2:33963)
0926 15:03:15.222989 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:15.222991 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:15.223003 (+ 12us) server_negotiation.cc:407] Connection header received
0926 15:03:15.223029 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:15.223030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:15.223042 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:15.223105 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:03:15.223262 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.223997 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.225610 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.226220 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.226300 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:15.226315 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:15.226549 (+ 234us) server_negotiation.cc:299] Negotiation successful
0926 15:03:15.226596 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260926 15:03:15.226723 24471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:15.222456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:46003 (local address 127.8.102.1:54843)
0926 15:03:15.222721 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:15.222734 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:03:15.222808 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:03:15.223016 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:03:15.223017 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:03:15.223019 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:03:15.223129 (+ 110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:15.223131 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.224120 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.224121 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:15.225482 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:15.225486 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.226370 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.226372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:15.226462 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:15.226466 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:03:15.226543 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 15:03:15.226599 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"mutex_wait_us":58,"thread_start_us":247,"threads_started":1}
I20260926 15:03:15.226716 24460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:15.222590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:54843 (local address 127.8.102.3:46003)
0926 15:03:15.222717 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:15.222720 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:15.222803 (+ 83us) server_negotiation.cc:407] Connection header received
0926 15:03:15.222879 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:15.222880 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:15.222893 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:15.222998 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:03:15.223242 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.224005 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.225611 (+ 1606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.226268 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.226329 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:15.226339 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:15.226575 (+ 236us) server_negotiation.cc:299] Negotiation successful
0926 15:03:15.226622 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 15:03:15.226744 24472 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:15.222828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:33963 (local address 127.8.102.1:36057)
0926 15:03:15.222917 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:15.222926 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 15:03:15.222966 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:03:15.223121 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:03:15.223122 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:03:15.223124 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:03:15.223189 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:15.223190 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.224140 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.224141 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:15.225481 (+ 1340us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:15.225486 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.226357 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.226359 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:15.226447 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:15.226452 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:03:15.226541 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 15:03:15.226597 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77,"mutex_wait_us":87,"thread_start_us":51,"threads_started":1}
I20260926 15:03:15.227253 24399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5096d6131649f3b8ff0568d5cffd3f" candidate_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ef46f3beb7049288f0dd6d2de848a71" is_pre_election: true
I20260926 15:03:15.227262 24265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5096d6131649f3b8ff0568d5cffd3f" candidate_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b339edec4d20486393117d522b04f410" is_pre_election: true
I20260926 15:03:15.227393 24399 raft_consensus.cc:2454] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf31f40b5b34e36b5840dead2e75f4f in term 0.
I20260926 15:03:15.227443 24265 raft_consensus.cc:2454] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf31f40b5b34e36b5840dead2e75f4f in term 0.
I20260926 15:03:15.227710 24065 leader_election.cc:302] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf31f40b5b34e36b5840dead2e75f4f, b339edec4d20486393117d522b04f410; no voters:
I20260926 15:03:15.227872 24468 raft_consensus.cc:2781] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:03:15.227953 24468 raft_consensus.cc:482] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:03:15.227991 24468 raft_consensus.cc:3037] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:03:15.228734 24468 raft_consensus.cc:504] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:15.228888 24468 leader_election.cc:288] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 1 election: Requested vote from peers 1ef46f3beb7049288f0dd6d2de848a71 (127.8.102.3:46003), b339edec4d20486393117d522b04f410 (127.8.102.2:33963)
I20260926 15:03:15.229175 24399 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5096d6131649f3b8ff0568d5cffd3f" candidate_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ef46f3beb7049288f0dd6d2de848a71"
I20260926 15:03:15.229247 24265 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6c5096d6131649f3b8ff0568d5cffd3f" candidate_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b339edec4d20486393117d522b04f410"
I20260926 15:03:15.229259 24399 raft_consensus.cc:3037] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:03:15.229348 24265 raft_consensus.cc:3037] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:03:15.230005 24399 raft_consensus.cc:2454] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf31f40b5b34e36b5840dead2e75f4f in term 1.
I20260926 15:03:15.230165 24265 raft_consensus.cc:2454] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf31f40b5b34e36b5840dead2e75f4f in term 1.
I20260926 15:03:15.230187 24065 leader_election.cc:302] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ef46f3beb7049288f0dd6d2de848a71, 9cf31f40b5b34e36b5840dead2e75f4f; no voters:
I20260926 15:03:15.230274 24468 raft_consensus.cc:2781] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:03:15.230419 24468 raft_consensus.cc:686] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 LEADER]: Becoming Leader. State: Replica: 9cf31f40b5b34e36b5840dead2e75f4f, State: Running, Role: LEADER
I20260926 15:03:15.230535 24468 consensus_queue.cc:237] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:15.231506 23985 catalog_manager.cc:5495] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cf31f40b5b34e36b5840dead2e75f4f (127.8.102.1). New cstate: current_term: 1 leader_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } health_report { overall_health: HEALTHY } } }
W20260926 15:03:15.244977 24446 tablet.cc:2369] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:03:15.260304 24458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:15.257597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42356 (local address 127.8.102.1:43347)
0926 15:03:15.257671 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:15.257673 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:15.257682 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:03:15.257733 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:15.257733 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:15.257752 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:15.257814 (+ 62us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:03:15.258025 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.258502 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.259365 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:15.259601 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:15.259681 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:15.259861 (+ 180us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:03:15.259974 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:03:15.260054 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:15.260143 (+ 89us) server_negotiation.cc:299] Negotiation successful
0926 15:03:15.260165 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:03:15.262437 24265 raft_consensus.cc:1261] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410 [term 1 FOLLOWER]: Refusing update from remote peer 9cf31f40b5b34e36b5840dead2e75f4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:03:15.262478 24399 raft_consensus.cc:1261] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71 [term 1 FOLLOWER]: Refusing update from remote peer 9cf31f40b5b34e36b5840dead2e75f4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:03:15.262854 24473 consensus_queue.cc:1035] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:03:15.263005 24468 consensus_queue.cc:1035] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:03:15.265786 24478 mvcc.cc:205] Tried to move back new op lower bound from 7333621740591566848 to 7333621740465082368. Current Snapshot: MvccSnapshot[applied={T|T < 7333621740591566848}]
I20260926 15:03:15.266004 24477 mvcc.cc:205] Tried to move back new op lower bound from 7333621740591566848 to 7333621740465082368. Current Snapshot: MvccSnapshot[applied={T|T < 7333621740591566848}]
I20260926 15:03:15.267966 23985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:03:15.268078 23985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34398:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:03:15.268864 23985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff: Sending DeleteTablet for 3 replicas of tablet 6c5096d6131649f3b8ff0568d5cffd3f
I20260926 15:03:15.269207 24379 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c5096d6131649f3b8ff0568d5cffd3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:03:15 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:58844
I20260926 15:03:15.269222 24111 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c5096d6131649f3b8ff0568d5cffd3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:03:15 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:42344
I20260926 15:03:15.269265 24245 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c5096d6131649f3b8ff0568d5cffd3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:03:15 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:58634
I20260926 15:03:15.269551 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 24048
I20260926 15:03:15.269558 24486 tablet_replica.cc:330] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f: stopping tablet replica
I20260926 15:03:15.269598 24488 tablet_replica.cc:330] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410: stopping tablet replica
I20260926 15:03:15.269663 24486 raft_consensus.cc:2229] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:03:15.269704 24488 raft_consensus.cc:2229] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:03:15.269760 24486 raft_consensus.cc:2258] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:03:15.269812 24488 raft_consensus.cc:2258] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:03:15.270159 24487 tablet_replica.cc:330] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71: stopping tablet replica
I20260926 15:03:15.270295 24487 raft_consensus.cc:2229] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:03:15.270371 24487 raft_consensus.cc:2258] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:03:15.270713 24488 ts_tablet_manager.cc:1905] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:03:15.270805 24487 ts_tablet_manager.cc:1905] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:03:15.272388 24487 ts_tablet_manager.cc:1918] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:03:15.272444 24487 log.cc:1196] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/6c5096d6131649f3b8ff0568d5cffd3f
I20260926 15:03:15.272708 24488 ts_tablet_manager.cc:1918] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:03:15.272758 24487 ts_tablet_manager.cc:1939] T 6c5096d6131649f3b8ff0568d5cffd3f P 1ef46f3beb7049288f0dd6d2de848a71: Deleting consensus metadata
I20260926 15:03:15.272780 24488 log.cc:1196] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/6c5096d6131649f3b8ff0568d5cffd3f
I20260926 15:03:15.273038 24488 ts_tablet_manager.cc:1939] T 6c5096d6131649f3b8ff0568d5cffd3f P b339edec4d20486393117d522b04f410: Deleting consensus metadata
I20260926 15:03:15.273150 23971 catalog_manager.cc:4841] TS 1ef46f3beb7049288f0dd6d2de848a71 (127.8.102.3:46003): tablet 6c5096d6131649f3b8ff0568d5cffd3f (table test-table [id=2434c9bbf5db4687b46a1e5e618b0a2f]) successfully deleted
I20260926 15:03:15.273607 23971 catalog_manager.cc:4841] TS b339edec4d20486393117d522b04f410 (127.8.102.2:33963): tablet 6c5096d6131649f3b8ff0568d5cffd3f (table test-table [id=2434c9bbf5db4687b46a1e5e618b0a2f]) successfully deleted
W20260926 15:03:15.276451 23970 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.102.1:43347 (error 108)
I20260926 15:03:15.276721 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 24182
I20260926 15:03:15.277479 24459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:15.276849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:43347 (local address 127.0.0.1:42368)
0926 15:03:15.277382 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:15.277435 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.1:43347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504}
W20260926 15:03:15.277644 23970 catalog_manager.cc:4584] TS 9cf31f40b5b34e36b5840dead2e75f4f (127.8.102.1:43347): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6c5096d6131649f3b8ff0568d5cffd3f: Network error: Client connection negotiation failed: client connection to 127.8.102.1:43347: connect: Connection refused (error 111)
I20260926 15:03:15.282363 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 24316
I20260926 15:03:15.286962 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 23954
I20260926 15:03:35.292687 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:36407
--webserver_interface=127.8.102.62
--webserver_port=34717
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:36407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:03:35.379640 24490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:03:35.379918 24490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:03:35.379966 24490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:03:35.381330 24490 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 15:03:35.381407 24490 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 15:03:35.381433 24490 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 15:03:35.381455 24490 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:03:35.381486 24490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:03:35.381522 24490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:03:35.381551 24490 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:03:35.381574 24490 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:03:35.383318 24490 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:36407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:36407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=34717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24490
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:03:35.383643 24490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:03:35.383910 24490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:03:35.393431 24499 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:03:35.393486 24490 server_base.cc:1034] running on GCE node
W20260926 15:03:35.393543 24496 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:03:35.393442 24497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:03:35.394197 24490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:03:35.394583 24490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:03:35.395820 24490 hybrid_clock.cc:648] HybridClock initialized: now 1790435015395797 us; error 45 us; skew 500 ppm
Sep 26 15:03:35 dist-test-slave-jnhs krb5kdc[23941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435015, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:03:35.398731 24490 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:03:35.399283 24490 webserver.cc:460] Webserver started at http://127.8.102.62:34717/ using document root <none> and password file <none>
I20260926 15:03:35.399533 24490 fs_manager.cc:359] Metadata directory not provided
I20260926 15:03:35.399591 24490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:03:35.401041 24490 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:03:35.401821 24506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:35.402088 24490 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:03:35.402196 24490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "14791678bc4b4a908632f5923634feff"
format_stamp: "Formatted at 2026-09-26 15:03:14 on dist-test-slave-jnhs"
server_key: "767991b628f7f2eced21d862f35c87d8"
server_key_iv: "e2f6abe6b4a4d1ba5702433af036abcf"
server_key_version: "encryptionkey@0"
I20260926 15:03:35.402751 24490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:03:35.441697 24490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:03:35.444298 24490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:03:35.444422 24490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:03:35.448622 24490 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:36407
I20260926 15:03:35.448716 24558 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:36407 every 8 connection(s)
I20260926 15:03:35.449904 24490 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:03:35.451736 24559 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:03:35.452554 24559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff: Bootstrap starting.
I20260926 15:03:35.459154 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 24490
I20260926 15:03:35.459450 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:43347
--local_ip_for_outbound_sockets=127.8.102.1
--tserver_master_addrs=127.8.102.62:36407
--webserver_port=34731
--webserver_interface=127.8.102.1
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:03:35.461879 24559 log.cc:824] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff: Log is configured to *not* fsync() on all Append() calls
I20260926 15:03:35.464371 24559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:03:35.464751 24559 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff: Bootstrap complete.
I20260926 15:03:35.467536 24559 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14791678bc4b4a908632f5923634feff" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36407 } }
I20260926 15:03:35.467796 24559 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14791678bc4b4a908632f5923634feff, State: Initialized, Role: FOLLOWER
I20260926 15:03:35.467993 24559 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14791678bc4b4a908632f5923634feff" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36407 } }
I20260926 15:03:35.468060 24559 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:03:35.468120 24559 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:03:35.468175 24559 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 1 FOLLOWER]: Advancing to term 2
I20260926 15:03:35.469339 24559 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14791678bc4b4a908632f5923634feff" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36407 } }
I20260926 15:03:35.469435 24559 leader_election.cc:302] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14791678bc4b4a908632f5923634feff; no voters:
I20260926 15:03:35.469671 24559 leader_election.cc:288] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [CANDIDATE]: Term 2 election: Requested vote from peers
I20260926 15:03:35.469719 24563 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 2 FOLLOWER]: Leader election won for term 2
I20260926 15:03:35.469898 24563 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 2 LEADER]: Becoming Leader. State: Replica: 14791678bc4b4a908632f5923634feff, State: Running, Role: LEADER
I20260926 15:03:35.469996 24559 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:03:35.470059 24563 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14791678bc4b4a908632f5923634feff" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36407 } }
I20260926 15:03:35.470405 24563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14791678bc4b4a908632f5923634feff. Latest consensus state: current_term: 2 leader_uuid: "14791678bc4b4a908632f5923634feff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14791678bc4b4a908632f5923634feff" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36407 } } }
I20260926 15:03:35.470522 24563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [sys.catalog]: This master's current role is: LEADER
I20260926 15:03:35.470409 24564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "14791678bc4b4a908632f5923634feff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14791678bc4b4a908632f5923634feff" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36407 } } }
I20260926 15:03:35.470769 24564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [sys.catalog]: This master's current role is: LEADER
I20260926 15:03:35.470770 24569 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:03:35.471817 24569 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:03:35.471951 24569 catalog_manager.cc:1259] Loaded cluster ID: 0accf826d80b411497c1d0b95062ad1a
I20260926 15:03:35.471998 24569 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:03:35.473268 24569 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:03:35.473474 24569 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff: Loaded TSK: 0
I20260926 15:03:35.479892 24569 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff: Generated new TSK 1
I20260926 15:03:35.480034 24569 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 15:03:35.565217 24561 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:03:35.565459 24561 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:03:35.565502 24561 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:03:35.566709 24561 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:03:35.566778 24561 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:03:35.566826 24561 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:03:35.568374 24561 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:43347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=34731
--tserver_master_addrs=127.8.102.62:36407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24561
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:03:35.568665 24561 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:03:35.568917 24561 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:03:35.575598 24586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:03:35.575634 24589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:03:35.575970 24587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:03:35.576169 24561 server_base.cc:1034] running on GCE node
I20260926 15:03:35.576380 24561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:03:35.576704 24561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:03:35.577906 24561 hybrid_clock.cc:648] HybridClock initialized: now 1790435015577891 us; error 31 us; skew 500 ppm
Sep 26 15:03:35 dist-test-slave-jnhs krb5kdc[23941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435015, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:03:35.580621 24561 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:03:35.581084 24561 webserver.cc:460] Webserver started at http://127.8.102.1:34731/ using document root <none> and password file <none>
I20260926 15:03:35.581332 24561 fs_manager.cc:359] Metadata directory not provided
I20260926 15:03:35.581389 24561 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:03:35.582882 24561 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:03:35.583644 24596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:35.583854 24561 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:03:35.583957 24561 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "9cf31f40b5b34e36b5840dead2e75f4f"
format_stamp: "Formatted at 2026-09-26 15:03:14 on dist-test-slave-jnhs"
server_key: "99643f6f2998539a8e74a1b6aee9e8fd"
server_key_iv: "6e375f1638b80e145d4b328e5359d180"
server_key_version: "encryptionkey@0"
I20260926 15:03:35.584391 24561 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:03:35.599604 24561 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:03:35.602003 24561 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:03:35.602228 24561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:03:35.602502 24561 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:03:35.603040 24603 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 15:03:35.604489 24561 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 15:03:35.604545 24561 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 15:03:35.604573 24561 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 15:03:35.605230 24561 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 15:03:35.605275 24561 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 15:03:35.605320 24603 tablet_bootstrap.cc:492] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f: Bootstrap starting.
I20260926 15:03:35.613363 24561 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:43347
I20260926 15:03:35.613400 24710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:43347 every 8 connection(s)
I20260926 15:03:35.613981 24561 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:03:35.615451 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 24561
I20260926 15:03:35.615805 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:33963
--local_ip_for_outbound_sockets=127.8.102.2
--tserver_master_addrs=127.8.102.62:36407
--webserver_port=38937
--webserver_interface=127.8.102.2
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:03:35.617231 24603 log.cc:824] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f: Log is configured to *not* fsync() on all Append() calls
I20260926 15:03:35.620229 24603 tablet_bootstrap.cc:492] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:03:35.620561 24603 tablet_bootstrap.cc:492] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f: Bootstrap complete.
I20260926 15:03:35.621867 24603 ts_tablet_manager.cc:1397] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.010s
I20260926 15:03:35.624117 24603 raft_consensus.cc:348] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:35.624610 24603 raft_consensus.cc:729] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf31f40b5b34e36b5840dead2e75f4f, State: Initialized, Role: FOLLOWER
I20260926 15:03:35.624837 24603 consensus_queue.cc:260] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:35.625213 24603 ts_tablet_manager.cc:1428] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
Sep 26 15:03:35 dist-test-slave-jnhs krb5kdc[23941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435015, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:03:35.632863 24713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:35.614841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:40319 (local address 127.8.102.62:36407)
0926 15:03:35.618022 (+ 3181us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:35.618032 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:35.618084 (+ 52us) server_negotiation.cc:407] Connection header received
0926 15:03:35.618235 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:35.618243 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:35.618308 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:35.618558 (+ 250us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:03:35.619435 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.620898 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.623369 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.624237 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.626491 (+ 2254us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:35.628026 (+ 1535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:03:35.628031 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:03:35.628032 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:03:35.628045 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:03:35.630299 (+ 2254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:35.631007 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:35.631011 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:35.631011 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:35.631172 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:35.631714 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:35.631716 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:35.631716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:35.632017 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:03:35.632116 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:35.632525 (+ 409us) server_negotiation.cc:299] Negotiation successful
0926 15:03:35.632674 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3116,"thread_start_us":80,"threads_started":1}
I20260926 15:03:35.632874 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:35.614831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.1:40319)
0926 15:03:35.615096 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:35.615138 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:03:35.615835 (+ 697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:03:35.618616 (+ 2781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:03:35.618621 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:03:35.619005 (+ 384us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:03:35.619281 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:35.619289 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.621107 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.621109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:35.622849 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:35.622854 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.624403 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.624405 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:35.624796 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:35.625657 (+ 861us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:03:35.625669 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:03:35.627884 (+ 2215us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:03:35.630423 (+ 2539us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:03:35.630433 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:03:35.630436 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:03:35.630893 (+ 457us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:03:35.631271 (+ 378us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:03:35.631272 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:03:35.631273 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:03:35.631607 (+ 334us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:03:35.632122 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:03:35.632130 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:03:35.632306 (+ 176us) client_negotiation.cc:769] Sending connection context
0926 15:03:35.632425 (+ 119us) client_negotiation.cc:240] Negotiation successful
0926 15:03:35.632652 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224,"thread_start_us":91,"threads_started":1}
I20260926 15:03:35.634079 24711 heartbeater.cc:347] Connected to a master server at 127.8.102.62:36407
I20260926 15:03:35.634189 24711 heartbeater.cc:464] Registering TS with master...
I20260926 15:03:35.634369 24711 heartbeater.cc:511] Master 127.8.102.62:36407 requested a full tablet report, sending...
I20260926 15:03:35.635111 24523 ts_manager.cc:194] Registered new tserver with Master: 9cf31f40b5b34e36b5840dead2e75f4f (127.8.102.1:43347)
I20260926 15:03:35.636245 24523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:40319
I20260926 15:03:35.637261 24711 heartbeater.cc:503] Master 127.8.102.62:36407 was elected leader, sending a full tablet report...
Sep 26 15:03:35 dist-test-slave-jnhs krb5kdc[23941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435015, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for kudu/127.8.102.1@KRBTEST.COM
I20260926 15:03:35.645305 24719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:35.636566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37256 (local address 127.8.102.1:43347)
0926 15:03:35.636741 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:35.636745 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:35.636754 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:03:35.636783 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:35.636785 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:35.636797 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:35.636878 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:03:35.637329 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.638162 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.638995 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.639298 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.639384 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:35.641700 (+ 2316us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:03:35.641709 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:03:35.641709 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:03:35.641715 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:03:35.643642 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:35.644169 (+ 527us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:35.644172 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:35.644173 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:35.644247 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:35.644571 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:35.644572 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:35.644572 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:35.644754 (+ 182us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:03:35.644860 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:35.645071 (+ 211us) server_negotiation.cc:299] Negotiation successful
0926 15:03:35.645182 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":91,"threads_started":1}
I20260926 15:03:35.645308 24718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:35.635787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:43347 (local address 127.0.0.1:37256)
0926 15:03:35.635979 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:35.636021 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:03:35.636088 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:03:35.636897 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:03:35.636904 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:03:35.637192 (+ 288us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:03:35.637241 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:35.637243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.638250 (+ 1007us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.638251 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:35.638913 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:35.638916 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.639382 (+ 466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.639383 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:35.639471 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:35.640020 (+ 549us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:03:35.640024 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:03:35.641574 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:03:35.643751 (+ 2177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:03:35.643761 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:03:35.643763 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:03:35.644056 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:03:35.644383 (+ 327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:03:35.644385 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:03:35.644385 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:03:35.644487 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:03:35.644838 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:03:35.644848 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:03:35.644914 (+ 66us) client_negotiation.cc:769] Sending connection context
0926 15:03:35.645016 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 15:03:35.645181 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":154,"thread_start_us":91,"threads_started":1}
I20260926 15:03:35.645859 24627 tablet_service.cc:1514] Processing DeleteTablet for tablet 6c5096d6131649f3b8ff0568d5cffd3f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:03:15 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:37256
I20260926 15:03:35.646512 24720 tablet_replica.cc:330] stopping tablet replica
I20260926 15:03:35.646678 24720 raft_consensus.cc:2229] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:03:35.646765 24720 raft_consensus.cc:2258] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:03:35.647148 24720 ts_tablet_manager.cc:1905] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:03:35.649449 24720 ts_tablet_manager.cc:1918] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:03:35.649571 24720 log.cc:1196] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/wal/wals/6c5096d6131649f3b8ff0568d5cffd3f
I20260926 15:03:35.649820 24720 ts_tablet_manager.cc:1939] T 6c5096d6131649f3b8ff0568d5cffd3f P 9cf31f40b5b34e36b5840dead2e75f4f: Deleting consensus metadata
I20260926 15:03:35.650431 24507 catalog_manager.cc:4841] TS 9cf31f40b5b34e36b5840dead2e75f4f (127.8.102.1:43347): tablet 6c5096d6131649f3b8ff0568d5cffd3f (table test-table [id=2434c9bbf5db4687b46a1e5e618b0a2f]) successfully deleted
W20260926 15:03:35.710172 24715 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:03:35.710407 24715 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:03:35.710433 24715 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:03:35.711721 24715 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:03:35.711776 24715 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:03:35.711804 24715 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:03:35.713385 24715 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:33963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=38937
--tserver_master_addrs=127.8.102.62:36407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24715
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:03:35.713779 24715 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:03:35.714040 24715 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:03:35.719550 24725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:03:35.719566 24728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:03:35.719537 24726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:03:35.720070 24715 server_base.cc:1034] running on GCE node
I20260926 15:03:35.720353 24715 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:03:35.720654 24715 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:03:35.721858 24715 hybrid_clock.cc:648] HybridClock initialized: now 1790435015721842 us; error 58 us; skew 500 ppm
Sep 26 15:03:35 dist-test-slave-jnhs krb5kdc[23941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435015, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:03:35.724923 24715 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:03:35.725485 24715 webserver.cc:460] Webserver started at http://127.8.102.2:38937/ using document root <none> and password file <none>
I20260926 15:03:35.725703 24715 fs_manager.cc:359] Metadata directory not provided
I20260926 15:03:35.725764 24715 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:03:35.727375 24715 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:03:35.728101 24735 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:35.728413 24715 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:03:35.728505 24715 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "b339edec4d20486393117d522b04f410"
format_stamp: "Formatted at 2026-09-26 15:03:14 on dist-test-slave-jnhs"
server_key: "68227588a6454cddf2400aa727e9a942"
server_key_iv: "fb7a6b9f88c55b4d547355cf7acc4e2b"
server_key_version: "encryptionkey@0"
I20260926 15:03:35.728889 24715 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:03:35.741671 24715 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:03:35.744346 24715 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:03:35.744482 24715 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:03:35.744693 24715 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:03:35.745110 24715 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:03:35.745153 24715 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:35.745178 24715 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:03:35.745193 24715 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:35.751811 24715 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:33963
I20260926 15:03:35.751857 24848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:33963 every 8 connection(s)
I20260926 15:03:35.752251 24715 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:03:35.755273 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 24715
I20260926 15:03:35.755614 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:46003
--local_ip_for_outbound_sockets=127.8.102.3
--tserver_master_addrs=127.8.102.62:36407
--webserver_port=38469
--webserver_interface=127.8.102.3
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 26 15:03:35 dist-test-slave-jnhs krb5kdc[23941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435015, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:03:35.764468 24713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:35.753146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:52581 (local address 127.8.102.62:36407)
0926 15:03:35.753244 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:35.753247 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:35.753373 (+ 126us) server_negotiation.cc:407] Connection header received
0926 15:03:35.754126 (+ 753us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:35.754127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:35.754142 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:35.754219 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:03:35.755580 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.756185 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.757693 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.758019 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.758136 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:35.760864 (+ 2728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:03:35.760867 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:03:35.760867 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:03:35.760876 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:03:35.762736 (+ 1860us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:35.763235 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:35.763236 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:35.763237 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:35.763304 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:35.763643 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:35.763644 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:35.763644 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:35.763815 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:03:35.763897 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:35.764221 (+ 324us) server_negotiation.cc:299] Negotiation successful
0926 15:03:35.764331 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260926 15:03:35.764549 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:35.753075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.2:52581)
0926 15:03:35.753243 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:35.753283 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0926 15:03:35.753988 (+ 705us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:03:35.754289 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:03:35.754292 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:03:35.754970 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:03:35.755325 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:35.755334 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.756301 (+ 967us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.756302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:35.757592 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:35.757595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.758190 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.758191 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:35.758389 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:35.759113 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:03:35.759124 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:03:35.760752 (+ 1628us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:03:35.762843 (+ 2091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:03:35.762850 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:03:35.762852 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:03:35.763148 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:03:35.763399 (+ 251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:03:35.763400 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:03:35.763400 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:03:35.763558 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:03:35.763910 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:03:35.763916 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:03:35.764060 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 15:03:35.764177 (+ 117us) client_negotiation.cc:240] Negotiation successful
0926 15:03:35.764315 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260926 15:03:35.765136 24849 heartbeater.cc:347] Connected to a master server at 127.8.102.62:36407
I20260926 15:03:35.765282 24849 heartbeater.cc:464] Registering TS with master...
I20260926 15:03:35.765518 24849 heartbeater.cc:511] Master 127.8.102.62:36407 requested a full tablet report, sending...
I20260926 15:03:35.766016 24523 ts_manager.cc:194] Registered new tserver with Master: b339edec4d20486393117d522b04f410 (127.8.102.2:33963)
I20260926 15:03:35.766578 24523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:52581
W20260926 15:03:35.840559 24852 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:03:35.840780 24852 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:03:35.840803 24852 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:03:35.842072 24852 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:03:35.842136 24852 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:03:35.842200 24852 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:03:35.843806 24852 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:46003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=38469
--tserver_master_addrs=127.8.102.62:36407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24852
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:03:35.844009 24852 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:03:35.844208 24852 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:03:35.849710 24860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:03:35.849694 24857 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:03:35.849745 24858 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:03:35.849910 24852 server_base.cc:1034] running on GCE node
I20260926 15:03:35.850129 24852 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:03:35.850355 24852 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:03:35.851540 24852 hybrid_clock.cc:648] HybridClock initialized: now 1790435015851532 us; error 38 us; skew 500 ppm
Sep 26 15:03:35 dist-test-slave-jnhs krb5kdc[23941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435015, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:03:35.854329 24852 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:03:35.854887 24852 webserver.cc:460] Webserver started at http://127.8.102.3:38469/ using document root <none> and password file <none>
I20260926 15:03:35.855124 24852 fs_manager.cc:359] Metadata directory not provided
I20260926 15:03:35.855209 24852 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:03:35.856693 24852 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:03:35.857429 24867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:35.857631 24852 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:03:35.857743 24852 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "1ef46f3beb7049288f0dd6d2de848a71"
format_stamp: "Formatted at 2026-09-26 15:03:14 on dist-test-slave-jnhs"
server_key: "310ac9bb844cc46ad996f211e24fd6f4"
server_key_iv: "1f945a0adee040a3bb88b30e6b70fea5"
server_key_version: "encryptionkey@0"
I20260926 15:03:35.858111 24852 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:03:35.883051 24852 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:03:35.885525 24852 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:03:35.885763 24852 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:03:35.886026 24852 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:03:35.886404 24852 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:03:35.886466 24852 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:35.886508 24852 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:03:35.886551 24852 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:35.893428 24852 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:46003
I20260926 15:03:35.893477 24980 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:46003 every 8 connection(s)
I20260926 15:03:35.893867 24852 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
Sep 26 15:03:35 dist-test-slave-jnhs krb5kdc[23941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435015, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:03:35.901825 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 24852
I20260926 15:03:35.904232 24713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:35.894670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:45033 (local address 127.8.102.62:36407)
0926 15:03:35.894732 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:35.894734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:35.894984 (+ 250us) server_negotiation.cc:407] Connection header received
0926 15:03:35.895565 (+ 581us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:35.895567 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:35.895578 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:35.895650 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:03:35.896644 (+ 994us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.897171 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.898177 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.898388 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.898450 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:35.900789 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:03:35.900791 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:03:35.900791 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:03:35.900798 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:03:35.902571 (+ 1773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:35.903106 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:35.903107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:35.903108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:35.903169 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:35.903541 (+ 372us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:35.903542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:35.903543 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:35.903684 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:03:35.903760 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:35.904046 (+ 286us) server_negotiation.cc:299] Negotiation successful
0926 15:03:35.904140 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 15:03:35.904323 24983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:35.894646 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.3:45033)
0926 15:03:35.894841 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:35.894880 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 15:03:35.895434 (+ 554us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:03:35.895709 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:03:35.895712 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:03:35.896330 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:03:35.896534 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:35.896539 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.897262 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.897263 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:35.898089 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:35.898092 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.898506 (+ 414us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.898507 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:35.898624 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:35.899283 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:03:35.899293 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:03:35.900681 (+ 1388us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:03:35.902678 (+ 1997us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:03:35.902686 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:03:35.902690 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:03:35.903009 (+ 319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:03:35.903246 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:03:35.903247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:03:35.903247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:03:35.903389 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:03:35.903757 (+ 368us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:03:35.903762 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:03:35.903891 (+ 129us) client_negotiation.cc:769] Sending connection context
0926 15:03:35.903986 (+ 95us) client_negotiation.cc:240] Negotiation successful
0926 15:03:35.904121 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":97,"threads_started":1}
I20260926 15:03:35.904822 24981 heartbeater.cc:347] Connected to a master server at 127.8.102.62:36407
I20260926 15:03:35.904925 24981 heartbeater.cc:464] Registering TS with master...
I20260926 15:03:35.905120 24981 heartbeater.cc:511] Master 127.8.102.62:36407 requested a full tablet report, sending...
I20260926 15:03:35.905548 24523 ts_manager.cc:194] Registered new tserver with Master: 1ef46f3beb7049288f0dd6d2de848a71 (127.8.102.3:46003)
I20260926 15:03:35.905941 24523 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:45033
I20260926 15:03:35.909636 24985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:35.902371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50802 (local address 127.8.102.62:36407)
0926 15:03:35.902513 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:35.902515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:35.902575 (+ 60us) server_negotiation.cc:407] Connection header received
0926 15:03:35.902712 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:35.902712 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:35.902723 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:35.902781 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:03:35.903533 (+ 752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.904217 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.905339 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.905638 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.905706 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:35.907066 (+ 1360us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:03:35.907067 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:03:35.907068 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:03:35.907074 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:03:35.908444 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:35.908810 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:35.908811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:35.908811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:35.908901 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:35.909192 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:35.909193 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:35.909193 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:35.909296 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:03:35.909417 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:35.909481 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 15:03:35.909544 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":85,"threads_started":1}
I20260926 15:03:35.910509 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:03:35.913468 24985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:35.910911 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50814 (local address 127.8.102.62:36407)
0926 15:03:35.910956 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:35.910958 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:35.911050 (+ 92us) server_negotiation.cc:407] Connection header received
0926 15:03:35.911102 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:35.911103 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:35.911135 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:35.911198 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:03:35.911401 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.911859 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.912669 (+ 810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.912853 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.912906 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:35.913190 (+ 284us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:03:35.913289 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:03:35.913355 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:35.913398 (+ 43us) server_negotiation.cc:299] Negotiation successful
0926 15:03:35.913417 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 15:03:35.913774 24523 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:50814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:03:35.915589 24523 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:03:35.917904 24627 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8fa211f6e0449bb8f8bf331e82ef43 (DEFAULT_TABLE table=test-table [id=6b26110a45994412851bf691ae6527b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:03:35.918141 24627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f8fa211f6e0449bb8f8bf331e82ef43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:03:35.919620 24603 tablet_bootstrap.cc:492] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f: Bootstrap starting.
I20260926 15:03:35.919988 24603 tablet_bootstrap.cc:654] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f: Neither blocks nor log segments found. Creating new log.
I20260926 15:03:35.920549 24603 tablet_bootstrap.cc:492] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f: No bootstrap required, opened a new log
I20260926 15:03:35.920614 24603 ts_tablet_manager.cc:1397] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260926 15:03:35.920943 24603 raft_consensus.cc:348] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:35.921029 24603 raft_consensus.cc:374] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:03:35.921084 24603 raft_consensus.cc:729] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf31f40b5b34e36b5840dead2e75f4f, State: Initialized, Role: FOLLOWER
I20260926 15:03:35.921139 24603 consensus_queue.cc:260] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:35.921241 24603 ts_tablet_manager.cc:1428] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20260926 15:03:35.922446 24988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:35.917783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59282 (local address 127.8.102.2:33963)
0926 15:03:35.918484 (+ 701us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:35.918489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:35.918498 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:03:35.918535 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:35.918537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:35.918552 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:35.918659 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:03:35.918774 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.919790 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.921229 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.921980 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.922093 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:35.922158 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:35.922313 (+ 155us) server_negotiation.cc:299] Negotiation successful
0926 15:03:35.922355 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":94,"threads_started":1}
I20260926 15:03:35.922448 24718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:35.917672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:33963 (local address 127.0.0.1:59282)
0926 15:03:35.917710 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:35.917722 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:03:35.917761 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:03:35.918668 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:03:35.918669 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:03:35.918671 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:03:35.918711 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:35.918713 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.919875 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.919876 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:35.921132 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:35.921135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.922117 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.922118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:35.922208 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:35.922231 (+ 23us) client_negotiation.cc:769] Sending connection context
0926 15:03:35.922281 (+ 50us) client_negotiation.cc:240] Negotiation successful
0926 15:03:35.922333 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20,"mutex_wait_us":42}
I20260926 15:03:35.922456 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:35.917966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39998 (local address 127.8.102.3:46003)
0926 15:03:35.918161 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:35.918166 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:35.918175 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:03:35.918233 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:35.918235 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:35.918252 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:35.918338 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:03:35.918496 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.919662 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.921285 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.921891 (+ 606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.922020 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:35.922105 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:35.922329 (+ 224us) server_negotiation.cc:299] Negotiation successful
0926 15:03:35.922369 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":169,"thread_start_us":94,"threads_started":1}
I20260926 15:03:35.922456 24987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:35.917776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:46003 (local address 127.0.0.1:39998)
0926 15:03:35.917879 (+ 103us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:35.917887 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 15:03:35.917925 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:03:35.918353 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:03:35.918354 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:03:35.918356 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:03:35.918408 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:35.918409 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.919799 (+ 1390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.919801 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:35.921132 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:35.921135 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:35.922177 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:35.922178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:35.922249 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:35.922253 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:03:35.922309 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 15:03:35.922352 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"mutex_wait_us":45,"thread_start_us":69,"threads_started":1}
I20260926 15:03:35.923194 24783 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8fa211f6e0449bb8f8bf331e82ef43 (DEFAULT_TABLE table=test-table [id=6b26110a45994412851bf691ae6527b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:03:35.923481 24783 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f8fa211f6e0449bb8f8bf331e82ef43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:03:35.926299 24991 tablet_bootstrap.cc:492] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410: Bootstrap starting.
I20260926 15:03:35.927117 24991 tablet_bootstrap.cc:654] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410: Neither blocks nor log segments found. Creating new log.
I20260926 15:03:35.927038 24915 tablet_service.cc:1467] Processing CreateTablet for tablet 8f8fa211f6e0449bb8f8bf331e82ef43 (DEFAULT_TABLE table=test-table [id=6b26110a45994412851bf691ae6527b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:03:35.927385 24915 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8f8fa211f6e0449bb8f8bf331e82ef43. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:03:35.927795 24991 log.cc:824] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410: Log is configured to *not* fsync() on all Append() calls
I20260926 15:03:35.928603 24991 tablet_bootstrap.cc:492] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410: No bootstrap required, opened a new log
I20260926 15:03:35.928664 24991 ts_tablet_manager.cc:1397] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 15:03:35.930142 24993 tablet_bootstrap.cc:492] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71: Bootstrap starting.
I20260926 15:03:35.930248 24991 raft_consensus.cc:348] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:35.930464 24991 raft_consensus.cc:374] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:03:35.930526 24991 raft_consensus.cc:729] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b339edec4d20486393117d522b04f410, State: Initialized, Role: FOLLOWER
I20260926 15:03:35.930686 24991 consensus_queue.cc:260] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:35.931032 24993 tablet_bootstrap.cc:654] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71: Neither blocks nor log segments found. Creating new log.
I20260926 15:03:35.931026 24991 ts_tablet_manager.cc:1428] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:03:35.931041 24849 heartbeater.cc:503] Master 127.8.102.62:36407 was elected leader, sending a full tablet report...
I20260926 15:03:35.931674 24993 log.cc:824] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71: Log is configured to *not* fsync() on all Append() calls
I20260926 15:03:35.932535 24993 tablet_bootstrap.cc:492] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71: No bootstrap required, opened a new log
I20260926 15:03:35.932606 24993 ts_tablet_manager.cc:1397] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 15:03:35.934114 24993 raft_consensus.cc:348] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:35.934253 24993 raft_consensus.cc:374] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:03:35.934275 24993 raft_consensus.cc:729] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1ef46f3beb7049288f0dd6d2de848a71, State: Initialized, Role: FOLLOWER
I20260926 15:03:35.934403 24993 consensus_queue.cc:260] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:35.934721 24993 ts_tablet_manager.cc:1428] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:03:35.934790 24981 heartbeater.cc:503] Master 127.8.102.62:36407 was elected leader, sending a full tablet report...
W20260926 15:03:36.002846 24850 tablet.cc:2369] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:03:36.100274 24717 raft_consensus.cc:482] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:03:36.100421 24717 raft_consensus.cc:504] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:36.100790 24717 leader_election.cc:288] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b339edec4d20486393117d522b04f410 (127.8.102.2:33963), 1ef46f3beb7049288f0dd6d2de848a71 (127.8.102.3:46003)
I20260926 15:03:36.103971 24988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:36.101121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:56875 (local address 127.8.102.2:33963)
0926 15:03:36.101173 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:36.101175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:36.101187 (+ 12us) server_negotiation.cc:407] Connection header received
0926 15:03:36.101277 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:36.101278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:36.101292 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:36.101354 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:03:36.101526 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:36.102076 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:36.103169 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:36.103578 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:36.103633 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:36.103643 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:36.103857 (+ 214us) server_negotiation.cc:299] Negotiation successful
0926 15:03:36.103901 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 15:03:36.103971 24714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:36.101007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:33963 (local address 127.8.102.1:56875)
0926 15:03:36.101069 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:36.101083 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:03:36.101177 (+ 94us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:03:36.101367 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:03:36.101368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:03:36.101370 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:03:36.101431 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:36.101433 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:36.102168 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:36.102169 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:36.103081 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:36.103083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:36.103658 (+ 575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:36.103659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:36.103726 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:36.103757 (+ 31us) client_negotiation.cc:769] Sending connection context
0926 15:03:36.103850 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 15:03:36.103896 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 15:03:36.104061 24989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:36.101253 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:43309 (local address 127.8.102.3:46003)
0926 15:03:36.101296 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:36.101297 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:36.101306 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:03:36.101368 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:36.101369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:36.101381 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:36.101438 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:03:36.101641 (+ 203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:36.102205 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:36.103249 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:36.103646 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:36.103715 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:36.103726 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:36.103927 (+ 201us) server_negotiation.cc:299] Negotiation successful
0926 15:03:36.103976 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 15:03:36.104281 24997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:36.101129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:46003 (local address 127.8.102.1:43309)
0926 15:03:36.101241 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:36.101252 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:03:36.101306 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:03:36.101462 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:03:36.101463 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:03:36.101483 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:03:36.101568 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:36.101570 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:36.102298 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:36.102299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:36.103153 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:36.103156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:36.103744 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:36.103745 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:36.103825 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:36.103830 (+ 5us) client_negotiation.cc:769] Sending connection context
0926 15:03:36.104150 (+ 320us) client_negotiation.cc:240] Negotiation successful
0926 15:03:36.104199 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":78,"threads_started":1}
I20260926 15:03:36.104346 24803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f8fa211f6e0449bb8f8bf331e82ef43" candidate_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b339edec4d20486393117d522b04f410" is_pre_election: true
I20260926 15:03:36.104562 24803 raft_consensus.cc:2454] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf31f40b5b34e36b5840dead2e75f4f in term 0.
I20260926 15:03:36.104806 24598 leader_election.cc:302] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9cf31f40b5b34e36b5840dead2e75f4f, b339edec4d20486393117d522b04f410; no voters:
I20260926 15:03:36.104964 24717 raft_consensus.cc:2781] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:03:36.105032 24717 raft_consensus.cc:482] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:03:36.105068 24717 raft_consensus.cc:3037] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:03:36.105049 24935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f8fa211f6e0449bb8f8bf331e82ef43" candidate_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ef46f3beb7049288f0dd6d2de848a71" is_pre_election: true
I20260926 15:03:36.105221 24935 raft_consensus.cc:2454] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9cf31f40b5b34e36b5840dead2e75f4f in term 0.
I20260926 15:03:36.106015 24717 raft_consensus.cc:504] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:36.106156 24717 leader_election.cc:288] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 1 election: Requested vote from peers b339edec4d20486393117d522b04f410 (127.8.102.2:33963), 1ef46f3beb7049288f0dd6d2de848a71 (127.8.102.3:46003)
I20260926 15:03:36.106338 24803 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f8fa211f6e0449bb8f8bf331e82ef43" candidate_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b339edec4d20486393117d522b04f410"
I20260926 15:03:36.106360 24935 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8f8fa211f6e0449bb8f8bf331e82ef43" candidate_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1ef46f3beb7049288f0dd6d2de848a71"
I20260926 15:03:36.106410 24803 raft_consensus.cc:3037] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:03:36.106448 24935 raft_consensus.cc:3037] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:03:36.107306 24803 raft_consensus.cc:2454] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf31f40b5b34e36b5840dead2e75f4f in term 1.
I20260926 15:03:36.107329 24935 raft_consensus.cc:2454] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9cf31f40b5b34e36b5840dead2e75f4f in term 1.
I20260926 15:03:36.107517 24598 leader_election.cc:302] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1ef46f3beb7049288f0dd6d2de848a71, 9cf31f40b5b34e36b5840dead2e75f4f; no voters:
I20260926 15:03:36.107631 24717 raft_consensus.cc:2781] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:03:36.107795 24717 raft_consensus.cc:686] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 LEADER]: Becoming Leader. State: Replica: 9cf31f40b5b34e36b5840dead2e75f4f, State: Running, Role: LEADER
I20260926 15:03:36.107882 24717 consensus_queue.cc:237] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:36.108788 24523 catalog_manager.cc:5495] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f reported cstate change: term changed from 0 to 1, leader changed from <none> to 9cf31f40b5b34e36b5840dead2e75f4f (127.8.102.1). New cstate: current_term: 1 leader_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } health_report { overall_health: HEALTHY } } }
W20260926 15:03:36.115005 24712 tablet.cc:2369] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:03:36.144443 24982 tablet.cc:2369] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:03:36.153959 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:36.151451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37270 (local address 127.8.102.1:43347)
0926 15:03:36.151602 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:36.151604 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:36.151611 (+ 7us) server_negotiation.cc:407] Connection header received
0926 15:03:36.151645 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:36.151646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:36.151664 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:36.151753 (+ 89us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:03:36.151894 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:36.152408 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:36.153181 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:36.153384 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:36.153478 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:36.153661 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:03:36.153712 (+ 51us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0926 15:03:36.153868 (+ 156us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37270: token expired
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":59,"threads_started":1}
W20260926 15:03:36.154035 25000 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37270: token expired
I20260926 15:03:36.160750 24985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:36.154360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50816 (local address 127.8.102.62:36407)
0926 15:03:36.154428 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:36.154430 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:36.154438 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:03:36.154461 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:36.154461 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:36.154469 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:36.154517 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:03:36.155180 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:36.155672 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:36.156462 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:36.156669 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:36.156736 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:36.157957 (+ 1221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:03:36.157959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:03:36.157959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:03:36.157966 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:03:36.159642 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:36.159966 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:36.159966 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:36.159967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:36.160017 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:36.160223 (+ 206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:36.160224 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:36.160224 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:36.160325 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:03:36.160403 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:36.160597 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 15:03:36.160681 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260926 15:03:36.164084 25000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:36.161718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37284 (local address 127.8.102.1:43347)
0926 15:03:36.161760 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:36.161762 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:36.161771 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:03:36.161807 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:36.161808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:36.161817 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:36.161864 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:03:36.162046 (+ 182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:36.162544 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:36.163403 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:36.163595 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:36.163653 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:36.163759 (+ 106us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:03:36.163852 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:03:36.163888 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:36.164006 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 15:03:36.164025 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 15:03:36.165673 24935 raft_consensus.cc:1261] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71 [term 1 FOLLOWER]: Refusing update from remote peer 9cf31f40b5b34e36b5840dead2e75f4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:03:36.165704 24803 raft_consensus.cc:1261] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410 [term 1 FOLLOWER]: Refusing update from remote peer 9cf31f40b5b34e36b5840dead2e75f4f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:03:36.166080 24998 consensus_queue.cc:1035] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:03:36.166205 24717 consensus_queue.cc:1035] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [LEADER]: Connected to new peer: Peer: permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:03:36.168648 25002 mvcc.cc:205] Tried to move back new op lower bound from 7333621826211614720 to 7333621825978568704. Current Snapshot: MvccSnapshot[applied={T|T < 7333621826211614720}]
I20260926 15:03:36.171100 24522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:03:36.171228 24522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50816:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:03:36.171984 24522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff: Sending DeleteTablet for 3 replicas of tablet 8f8fa211f6e0449bb8f8bf331e82ef43
I20260926 15:03:36.172293 24627 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8fa211f6e0449bb8f8bf331e82ef43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:03:36 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:37256
I20260926 15:03:36.172312 24783 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8fa211f6e0449bb8f8bf331e82ef43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:03:36 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:59282
I20260926 15:03:36.172319 24915 tablet_service.cc:1514] Processing DeleteTablet for tablet 8f8fa211f6e0449bb8f8bf331e82ef43 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:03:36 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:39998
I20260926 15:03:36.172540 25012 tablet_replica.cc:330] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410: stopping tablet replica
I20260926 15:03:36.172673 25012 raft_consensus.cc:2229] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:03:36.172744 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 24561
I20260926 15:03:36.172838 25012 raft_consensus.cc:2258] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:03:36.173312 25013 tablet_replica.cc:330] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71: stopping tablet replica
I20260926 15:03:36.173445 25013 raft_consensus.cc:2229] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:03:36.173549 25013 raft_consensus.cc:2258] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:03:36.174012 25012 ts_tablet_manager.cc:1905] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:03:36.175452 25013 ts_tablet_manager.cc:1905] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:03:36.176141 25012 ts_tablet_manager.cc:1918] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:03:36.176219 25012 log.cc:1196] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/8f8fa211f6e0449bb8f8bf331e82ef43
I20260926 15:03:36.176512 25012 ts_tablet_manager.cc:1939] T 8f8fa211f6e0449bb8f8bf331e82ef43 P b339edec4d20486393117d522b04f410: Deleting consensus metadata
I20260926 15:03:36.176779 25013 ts_tablet_manager.cc:1918] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:03:36.176867 25013 log.cc:1196] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/8f8fa211f6e0449bb8f8bf331e82ef43
I20260926 15:03:36.176934 24508 catalog_manager.cc:4841] TS b339edec4d20486393117d522b04f410 (127.8.102.2:33963): tablet 8f8fa211f6e0449bb8f8bf331e82ef43 (table test-table [id=6b26110a45994412851bf691ae6527b5]) successfully deleted
I20260926 15:03:36.177172 25013 ts_tablet_manager.cc:1939] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 1ef46f3beb7049288f0dd6d2de848a71: Deleting consensus metadata
I20260926 15:03:36.177753 24508 catalog_manager.cc:4841] TS 1ef46f3beb7049288f0dd6d2de848a71 (127.8.102.3:46003): tablet 8f8fa211f6e0449bb8f8bf331e82ef43 (table test-table [id=6b26110a45994412851bf691ae6527b5]) successfully deleted
W20260926 15:03:36.179281 24507 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.102.1:43347 (error 108)
I20260926 15:03:36.179471 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 24715
I20260926 15:03:36.180279 24987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:36.179626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:43347 (local address 127.0.0.1:37286)
0926 15:03:36.180207 (+ 581us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:36.180249 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.1:43347: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":560}
W20260926 15:03:36.180419 24507 catalog_manager.cc:4584] TS 9cf31f40b5b34e36b5840dead2e75f4f (127.8.102.1:43347): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8f8fa211f6e0449bb8f8bf331e82ef43: Network error: Client connection negotiation failed: client connection to 127.8.102.1:43347: connect: Connection refused (error 111)
I20260926 15:03:36.185401 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 24852
I20260926 15:03:36.190039 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 24490
I20260926 15:03:56.195649 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:36407
--webserver_interface=127.8.102.62
--webserver_port=34717
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:36407
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:03:56.279435 25015 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:03:56.279692 25015 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:03:56.279731 25015 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:03:56.280932 25015 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260926 15:03:56.281013 25015 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260926 15:03:56.281035 25015 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260926 15:03:56.281064 25015 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:03:56.281097 25015 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:03:56.281127 25015 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:03:56.281157 25015 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:03:56.281184 25015 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:03:56.282799 25015 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:36407
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:36407
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=34717
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25015
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:03:56.283090 25015 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:03:56.283322 25015 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:03:56.288756 25020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:03:56.288769 25023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:03:56.288777 25015 server_base.cc:1034] running on GCE node
W20260926 15:03:56.288813 25021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:03:56.289265 25015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:03:56.289594 25015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:03:56.290776 25015 hybrid_clock.cc:648] HybridClock initialized: now 1790435036290755 us; error 46 us; skew 500 ppm
Sep 26 15:03:56 dist-test-slave-jnhs krb5kdc[23941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435036, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:03:56.293499 25015 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:03:56.293982 25015 webserver.cc:460] Webserver started at http://127.8.102.62:34717/ using document root <none> and password file <none>
I20260926 15:03:56.294245 25015 fs_manager.cc:359] Metadata directory not provided
I20260926 15:03:56.294296 25015 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:03:56.295785 25015 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:03:56.296578 25030 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:56.296772 25015 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:03:56.296880 25015 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "14791678bc4b4a908632f5923634feff"
format_stamp: "Formatted at 2026-09-26 15:03:14 on dist-test-slave-jnhs"
server_key: "767991b628f7f2eced21d862f35c87d8"
server_key_iv: "e2f6abe6b4a4d1ba5702433af036abcf"
server_key_version: "encryptionkey@0"
I20260926 15:03:56.297274 25015 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:03:56.308604 25015 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:03:56.310788 25015 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:03:56.310964 25015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:03:56.315289 25015 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:36407
I20260926 15:03:56.315358 25082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:36407 every 8 connection(s)
I20260926 15:03:56.315752 25015 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:03:56.317250 25083 sys_catalog.cc:263] Verifying existing consensus state
I20260926 15:03:56.317879 25083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff: Bootstrap starting.
I20260926 15:03:56.321220 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 25015
I20260926 15:03:56.321514 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:43347
--local_ip_for_outbound_sockets=127.8.102.1
--tserver_master_addrs=127.8.102.62:36407
--webserver_port=34731
--webserver_interface=127.8.102.1
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:03:56.326485 25083 log.cc:824] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff: Log is configured to *not* fsync() on all Append() calls
I20260926 15:03:56.328842 25083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:03:56.329186 25083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff: Bootstrap complete.
I20260926 15:03:56.331543 25083 raft_consensus.cc:348] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14791678bc4b4a908632f5923634feff" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36407 } }
I20260926 15:03:56.331796 25083 raft_consensus.cc:729] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 14791678bc4b4a908632f5923634feff, State: Initialized, Role: FOLLOWER
I20260926 15:03:56.332012 25083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14791678bc4b4a908632f5923634feff" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36407 } }
I20260926 15:03:56.332113 25083 raft_consensus.cc:388] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:03:56.332167 25083 raft_consensus.cc:482] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:03:56.332214 25083 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 2 FOLLOWER]: Advancing to term 3
I20260926 15:03:56.333214 25083 raft_consensus.cc:504] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14791678bc4b4a908632f5923634feff" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36407 } }
I20260926 15:03:56.333335 25083 leader_election.cc:302] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 14791678bc4b4a908632f5923634feff; no voters:
I20260926 15:03:56.333561 25083 leader_election.cc:288] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [CANDIDATE]: Term 3 election: Requested vote from peers
I20260926 15:03:56.333603 25087 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 3 FOLLOWER]: Leader election won for term 3
I20260926 15:03:56.333720 25087 raft_consensus.cc:686] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [term 3 LEADER]: Becoming Leader. State: Replica: 14791678bc4b4a908632f5923634feff, State: Running, Role: LEADER
I20260926 15:03:56.333837 25083 sys_catalog.cc:564] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:03:56.333815 25087 consensus_queue.cc:237] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14791678bc4b4a908632f5923634feff" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36407 } }
I20260926 15:03:56.334214 25088 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "14791678bc4b4a908632f5923634feff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14791678bc4b4a908632f5923634feff" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36407 } } }
I20260926 15:03:56.334301 25088 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [sys.catalog]: This master's current role is: LEADER
I20260926 15:03:56.334537 25089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 14791678bc4b4a908632f5923634feff. Latest consensus state: current_term: 3 leader_uuid: "14791678bc4b4a908632f5923634feff" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "14791678bc4b4a908632f5923634feff" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36407 } } }
I20260926 15:03:56.334616 25094 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:03:56.334658 25089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff [sys.catalog]: This master's current role is: LEADER
I20260926 15:03:56.335538 25094 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:03:56.335657 25094 catalog_manager.cc:1259] Loaded cluster ID: 0accf826d80b411497c1d0b95062ad1a
I20260926 15:03:56.335702 25094 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:03:56.336822 25094 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:03:56.337023 25094 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff: Loaded TSK: 0
I20260926 15:03:56.337064 25094 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 14791678bc4b4a908632f5923634feff: Loaded TSK: 1
F20260926 15:03:56.337289 25094 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f1ed91109ad google::LogMessage::Fail() at ??:0
@ 0x7f1ed911388a google::LogMessage::SendToLog() at ??:0
@ 0x7f1ed91104de google::LogMessage::Flush() at ??:0
@ 0x7f1ed9111a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f1edb87d104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f1edb87ad41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f1ee106a1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f1ee106a48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f1ee106a594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f1ee1030c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f1ee103aad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f1edafde0d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f1edafd3954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f1edbe376db start_thread at ??:0
@ 0x7f1ed7c0e61f clone at ??:0
W20260926 15:03:56.441478 25085 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:03:56.441797 25085 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:03:56.441838 25085 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:03:56.443732 25085 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:03:56.443796 25085 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:03:56.443841 25085 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:03:56.446025 25085 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:43347
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=34731
--tserver_master_addrs=127.8.102.62:36407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25085
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:03:56.446287 25085 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:03:56.446568 25085 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:03:56.452085 25125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:03:56.452122 25124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:03:56.452327 25127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:03:56.452226 25085 server_base.cc:1034] running on GCE node
I20260926 15:03:56.452613 25085 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:03:56.452877 25085 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:03:56.454054 25085 hybrid_clock.cc:648] HybridClock initialized: now 1790435036454039 us; error 31 us; skew 500 ppm
Sep 26 15:03:56 dist-test-slave-jnhs krb5kdc[23941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435036, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:03:56.456912 25085 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:03:56.457433 25085 webserver.cc:460] Webserver started at http://127.8.102.1:34731/ using document root <none> and password file <none>
I20260926 15:03:56.457700 25085 fs_manager.cc:359] Metadata directory not provided
I20260926 15:03:56.457757 25085 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:03:56.459563 25085 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.000s
I20260926 15:03:56.460357 25134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:56.460562 25085 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:03:56.460659 25085 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "9cf31f40b5b34e36b5840dead2e75f4f"
format_stamp: "Formatted at 2026-09-26 15:03:14 on dist-test-slave-jnhs"
server_key: "99643f6f2998539a8e74a1b6aee9e8fd"
server_key_iv: "6e375f1638b80e145d4b328e5359d180"
server_key_version: "encryptionkey@0"
I20260926 15:03:56.461109 25085 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:03:56.484705 25085 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:03:56.487457 25085 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:03:56.487609 25085 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:03:56.487879 25085 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:03:56.488415 25141 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260926 15:03:56.490108 25085 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260926 15:03:56.490168 25085 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260926 15:03:56.490206 25085 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260926 15:03:56.491109 25085 ts_tablet_manager.cc:610] Registered 1 tablets
I20260926 15:03:56.491156 25085 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260926 15:03:56.491251 25141 tablet_bootstrap.cc:492] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f: Bootstrap starting.
I20260926 15:03:56.498333 25085 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:43347
I20260926 15:03:56.498610 25248 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:43347 every 8 connection(s)
I20260926 15:03:56.498813 25085 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:03:56.502388 25141 log.cc:824] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f: Log is configured to *not* fsync() on all Append() calls
I20260926 15:03:56.504729 25141 tablet_bootstrap.cc:492] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260926 15:03:56.505085 25141 tablet_bootstrap.cc:492] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f: Bootstrap complete.
I20260926 15:03:56.506008 25141 ts_tablet_manager.cc:1397] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.011s
I20260926 15:03:56.507102 25141 raft_consensus.cc:348] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:56.507365 25141 raft_consensus.cc:729] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cf31f40b5b34e36b5840dead2e75f4f, State: Initialized, Role: FOLLOWER
I20260926 15:03:56.507449 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 25085
I20260926 15:03:56.507532 25141 consensus_queue.cc:260] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:56.507743 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:33963
--local_ip_for_outbound_sockets=127.8.102.2
--tserver_master_addrs=127.8.102.62:36407
--webserver_port=38937
--webserver_interface=127.8.102.2
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:03:56.507815 25141 ts_tablet_manager.cc:1428] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f: Time spent starting tablet: real 0.002s user 0.000s sys 0.005s
W20260926 15:03:56.569654 25251 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:03:56.499571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.1:42569)
0926 15:03:56.499827 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:56.499869 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:03:56.500476 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:03:56.569419 (+ 68943us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":216,"thread_start_us":91,"threads_started":1}
W20260926 15:03:56.569887 25249 heartbeater.cc:644] Failed to heartbeat to 127.8.102.62:36407 (0 consecutive failures): Network error: Failed to ping master at 127.8.102.62:36407: Client connection negotiation failed: client connection to 127.8.102.62:36407: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260926 15:03:56.570242 25251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:56.570116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.1:54273)
0926 15:03:56.570173 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:56.570212 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 15:03:56.570621 25251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:56.570534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.1:52617)
0926 15:03:56.570584 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:56.570606 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 15:03:56.570767 25249 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260926 15:03:56.600250 25254 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:03:56.600461 25254 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:03:56.600485 25254 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:03:56.601605 25254 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:03:56.601645 25254 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:03:56.601672 25254 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:03:56.603304 25254 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:33963
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=38937
--tserver_master_addrs=127.8.102.62:36407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25254
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:03:56.603504 25254 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:03:56.603681 25254 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:03:56.609015 25259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:03:56.608989 25262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:03:56.608999 25260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:03:56.609210 25254 server_base.cc:1034] running on GCE node
I20260926 15:03:56.609423 25254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:03:56.609678 25254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:03:56.610877 25254 hybrid_clock.cc:648] HybridClock initialized: now 1790435036610852 us; error 41 us; skew 500 ppm
Sep 26 15:03:56 dist-test-slave-jnhs krb5kdc[23941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435036, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:03:56.614161 25254 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:03:56.614729 25254 webserver.cc:460] Webserver started at http://127.8.102.2:38937/ using document root <none> and password file <none>
I20260926 15:03:56.614964 25254 fs_manager.cc:359] Metadata directory not provided
I20260926 15:03:56.615015 25254 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:03:56.616441 25254 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 15:03:56.617036 25269 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:56.617214 25254 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 15:03:56.617305 25254 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "b339edec4d20486393117d522b04f410"
format_stamp: "Formatted at 2026-09-26 15:03:14 on dist-test-slave-jnhs"
server_key: "68227588a6454cddf2400aa727e9a942"
server_key_iv: "fb7a6b9f88c55b4d547355cf7acc4e2b"
server_key_version: "encryptionkey@0"
I20260926 15:03:56.617656 25254 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:03:56.629875 25254 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:03:56.631798 25254 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:03:56.631958 25254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:03:56.632196 25254 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:03:56.632509 25254 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:03:56.632565 25254 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:56.632606 25254 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:03:56.632635 25254 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:56.638590 25254 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:33963
I20260926 15:03:56.638657 25382 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:33963 every 8 connection(s)
I20260926 15:03:56.638996 25254 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:03:56.640013 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:56.639644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.2:33659)
0926 15:03:56.639798 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:56.639871 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":70,"threads_started":1}
W20260926 15:03:56.640435 25383 heartbeater.cc:644] Failed to heartbeat to 127.8.102.62:36407 (0 consecutive failures): Network error: Failed to ping master at 127.8.102.62:36407: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
I20260926 15:03:56.640729 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:56.640637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.2:41811)
0926 15:03:56.640683 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:56.640711 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260926 15:03:56.641026 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:56.640955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.2:38779)
0926 15:03:56.640990 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:56.641011 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260926 15:03:56.641157 25383 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 15:03:56.643673 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 25254
I20260926 15:03:56.643954 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:46003
--local_ip_for_outbound_sockets=127.8.102.3
--tserver_master_addrs=127.8.102.62:36407
--webserver_port=38469
--webserver_interface=127.8.102.3
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:44065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:03:56.727815 25386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:03:56.728189 25386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:03:56.728231 25386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:03:56.730079 25386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:03:56.730134 25386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:03:56.730176 25386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:03:56.732525 25386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:44065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:46003
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=38469
--tserver_master_addrs=127.8.102.62:36407
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25386
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:03:56.732802 25386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:03:56.733067 25386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:03:56.738929 25394 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:03:56.738899 25392 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:03:56.738899 25386 server_base.cc:1034] running on GCE node
W20260926 15:03:56.738916 25391 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:03:56.739408 25386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:03:56.739699 25386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:03:56.740929 25386 hybrid_clock.cc:648] HybridClock initialized: now 1790435036740912 us; error 26 us; skew 500 ppm
Sep 26 15:03:56 dist-test-slave-jnhs krb5kdc[23941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435036, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:03:56.743492 25386 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:03:56.743927 25386 webserver.cc:460] Webserver started at http://127.8.102.3:38469/ using document root <none> and password file <none>
I20260926 15:03:56.744194 25386 fs_manager.cc:359] Metadata directory not provided
I20260926 15:03:56.744266 25386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:03:56.745759 25386 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:03:56.746490 25401 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:56.746678 25386 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:03:56.746783 25386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "1ef46f3beb7049288f0dd6d2de848a71"
format_stamp: "Formatted at 2026-09-26 15:03:14 on dist-test-slave-jnhs"
server_key: "310ac9bb844cc46ad996f211e24fd6f4"
server_key_iv: "1f945a0adee040a3bb88b30e6b70fea5"
server_key_version: "encryptionkey@0"
I20260926 15:03:56.747151 25386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:03:56.756278 25386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:03:56.758391 25386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:03:56.758569 25386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:03:56.758836 25386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:03:56.759146 25386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:03:56.759197 25386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:56.759238 25386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:03:56.759269 25386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:03:56.765286 25386 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:46003
I20260926 15:03:56.765332 25514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:46003 every 8 connection(s)
I20260926 15:03:56.765683 25386 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:03:56.766796 25517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:56.766328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.3:41901)
0926 15:03:56.766476 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:56.766567 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":73,"threads_started":1}
W20260926 15:03:56.767184 25515 heartbeater.cc:644] Failed to heartbeat to 127.8.102.62:36407 (0 consecutive failures): Network error: Failed to ping master at 127.8.102.62:36407: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
I20260926 15:03:56.767464 25517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:56.767370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.3:55261)
0926 15:03:56.767417 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:56.767442 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260926 15:03:56.767733 25517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:56.767653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.3:35999)
0926 15:03:56.767693 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:56.767715 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260926 15:03:56.767835 25515 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260926 15:03:56.775338 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 25386
I20260926 15:03:57.571372 25519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:57.571116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.1:54785)
0926 15:03:57.571264 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:57.571310 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":63,"threads_started":1}
I20260926 15:03:57.641721 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:57.641501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.2:60865)
0926 15:03:57.641622 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:57.641667 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":52,"threads_started":1}
I20260926 15:03:57.768438 25521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:57.768216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.3:43969)
0926 15:03:57.768345 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:57.768387 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":56,"threads_started":1}
I20260926 15:03:58.276559 25522 raft_consensus.cc:482] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:03:58.276677 25522 raft_consensus.cc:504] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:03:58.277086 25522 leader_election.cc:288] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b339edec4d20486393117d522b04f410 (127.8.102.2:33963), 1ef46f3beb7049288f0dd6d2de848a71 (127.8.102.3:46003)
Sep 26 15:03:58 dist-test-slave-jnhs krb5kdc[23941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435036, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.2@KRBTEST.COM
Sep 26 15:03:58 dist-test-slave-jnhs krb5kdc[23941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435036, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.3@KRBTEST.COM
I20260926 15:03:58.288251 25524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:58.277405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:56453 (local address 127.8.102.2:33963)
0926 15:03:58.277651 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:58.277663 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:58.277691 (+ 28us) server_negotiation.cc:407] Connection header received
0926 15:03:58.278153 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:58.278156 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:58.278202 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:58.278366 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:03:58.279261 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:58.280153 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:58.281186 (+ 1033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:58.281591 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:58.281715 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:58.284397 (+ 2682us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:03:58.284401 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:03:58.284402 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:03:58.284413 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:03:58.286344 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:58.286946 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:58.286949 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:58.286950 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:58.287050 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:58.287336 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:58.287337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:58.287337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:58.287593 (+ 256us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:03:58.287697 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:58.288000 (+ 303us) server_negotiation.cc:299] Negotiation successful
0926 15:03:58.288133 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":202,"thread_start_us":60,"threads_started":1}
I20260926 15:03:58.288249 25523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:58.277272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:33963 (local address 127.8.102.1:56453)
0926 15:03:58.277421 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:58.277437 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:03:58.277516 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:03:58.278408 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:03:58.278411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:03:58.278976 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:03:58.279141 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:58.279147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:58.280266 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:58.280267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:58.281082 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:58.281084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:58.281686 (+ 602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:58.281687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:58.281804 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:58.282480 (+ 676us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:03:58.282484 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:03:58.284220 (+ 1736us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:03:58.286458 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:03:58.286466 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:03:58.286468 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:03:58.286857 (+ 389us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:03:58.287132 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:03:58.287133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:03:58.287133 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:03:58.287262 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:03:58.287676 (+ 414us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:03:58.287683 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:03:58.287826 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 15:03:58.287932 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 15:03:58.288153 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"mutex_wait_us":52,"thread_start_us":92,"threads_started":1}
I20260926 15:03:58.288275 25526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:58.277595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:49017 (local address 127.8.102.3:46003)
0926 15:03:58.277759 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0926 15:03:58.277767 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:03:58.277785 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:03:58.278196 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:03:58.278198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:03:58.278236 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:03:58.278380 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:03:58.279277 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:58.280152 (+ 875us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:58.281187 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:03:58.281596 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:03:58.281715 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:58.284593 (+ 2878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:03:58.284598 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:03:58.284598 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:03:58.284609 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:03:58.286404 (+ 1795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:58.286958 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:58.286961 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:58.286961 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:58.287058 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:03:58.287339 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:03:58.287340 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:03:58.287341 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:03:58.287606 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:03:58.287700 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 15:03:58.287999 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 15:03:58.288135 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":73,"threads_started":1}
I20260926 15:03:58.289233 25525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:58.277492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:46003 (local address 127.8.102.1:49017)
0926 15:03:58.277569 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:58.277578 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 15:03:58.277626 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:03:58.278408 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:03:58.278411 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:03:58.278921 (+ 510us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:03:58.279141 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:58.279147 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:58.280264 (+ 1117us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:58.280265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:58.281082 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:03:58.281084 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:03:58.281688 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:03:58.281689 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:03:58.281805 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:03:58.282454 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:03:58.282463 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:03:58.284466 (+ 2003us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:03:58.286510 (+ 2044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:03:58.286511 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:03:58.286512 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:03:58.286870 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:03:58.287132 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:03:58.287133 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:03:58.287134 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:03:58.287264 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:03:58.287683 (+ 419us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:03:58.287684 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:03:58.287826 (+ 142us) client_negotiation.cc:769] Sending connection context
0926 15:03:58.287932 (+ 106us) client_negotiation.cc:240] Negotiation successful
0926 15:03:58.288242 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62,"mutex_wait_us":184,"thread_start_us":41,"threads_started":1}
W20260926 15:03:58.289278 25136 leader_election.cc:341] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b339edec4d20486393117d522b04f410 (127.8.102.2:33963): Not found: Tablet not found: 8f8fa211f6e0449bb8f8bf331e82ef43
W20260926 15:03:58.290192 25136 leader_election.cc:341] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ef46f3beb7049288f0dd6d2de848a71 (127.8.102.3:46003): Not found: Tablet not found: 8f8fa211f6e0449bb8f8bf331e82ef43
I20260926 15:03:58.290241 25136 leader_election.cc:302] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9cf31f40b5b34e36b5840dead2e75f4f; no voters: 1ef46f3beb7049288f0dd6d2de848a71, b339edec4d20486393117d522b04f410
I20260926 15:03:58.290356 25522 raft_consensus.cc:2726] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:03:58.572062 25525 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:58.571907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.1:45093)
0926 15:03:58.571968 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:58.572011 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 15:03:58.642418 25527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:58.642200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.2:33515)
0926 15:03:58.642323 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:58.642372 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":60,"threads_started":1}
I20260926 15:03:58.769234 25528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:58.768957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.3:53013)
0926 15:03:58.769135 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:58.769187 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":131,"threads_started":1}
I20260926 15:03:59.572779 25529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:59.572548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.1:34519)
0926 15:03:59.572688 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:59.572730 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":69,"threads_started":1}
I20260926 15:03:59.643251 25530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:59.642980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.2:60417)
0926 15:03:59.643139 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:59.643179 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":143,"thread_start_us":74,"threads_started":1}
I20260926 15:03:59.769903 25531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:03:59.769701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.3:39231)
0926 15:03:59.769822 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0926 15:03:59.769857 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":59,"threads_started":1}
I20260926 15:04:00.075248 25532 raft_consensus.cc:482] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:04:00.075351 25532 raft_consensus.cc:504] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:04:00.075553 25532 leader_election.cc:288] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b339edec4d20486393117d522b04f410 (127.8.102.2:33963), 1ef46f3beb7049288f0dd6d2de848a71 (127.8.102.3:46003)
W20260926 15:04:00.076081 25136 leader_election.cc:341] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b339edec4d20486393117d522b04f410 (127.8.102.2:33963): Not found: Tablet not found: 8f8fa211f6e0449bb8f8bf331e82ef43
W20260926 15:04:00.076153 25136 leader_election.cc:341] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ef46f3beb7049288f0dd6d2de848a71 (127.8.102.3:46003): Not found: Tablet not found: 8f8fa211f6e0449bb8f8bf331e82ef43
I20260926 15:04:00.076206 25136 leader_election.cc:302] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9cf31f40b5b34e36b5840dead2e75f4f; no voters: 1ef46f3beb7049288f0dd6d2de848a71, b339edec4d20486393117d522b04f410
I20260926 15:04:00.076340 25532 raft_consensus.cc:2726] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:04:00.573693 25533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:00.573370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.1:41457)
0926 15:04:00.573561 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:00.573619 (+ 58us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":72,"threads_started":1}
I20260926 15:04:00.643944 25534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:00.643714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.2:42087)
0926 15:04:00.643853 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:00.643896 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":63,"threads_started":1}
I20260926 15:04:00.770726 25535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:00.770405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.3:51719)
0926 15:04:00.770569 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:00.770609 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":76,"threads_started":1}
I20260926 15:04:01.574708 25536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:01.574389 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.1:35175)
0926 15:04:01.574558 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:01.574607 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":65,"threads_started":1}
I20260926 15:04:01.644773 25537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:01.644504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.2:44241)
0926 15:04:01.644677 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:01.644724 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":66,"threads_started":1}
I20260926 15:04:01.771587 25538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:01.771352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.3:51601)
0926 15:04:01.771499 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:01.771541 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":76,"threads_started":1}
I20260926 15:04:02.408721 25539 raft_consensus.cc:482] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:04:02.408823 25539 raft_consensus.cc:504] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:04:02.408946 25539 leader_election.cc:288] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b339edec4d20486393117d522b04f410 (127.8.102.2:33963), 1ef46f3beb7049288f0dd6d2de848a71 (127.8.102.3:46003)
W20260926 15:04:02.409435 25136 leader_election.cc:341] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b339edec4d20486393117d522b04f410 (127.8.102.2:33963): Not found: Tablet not found: 8f8fa211f6e0449bb8f8bf331e82ef43
W20260926 15:04:02.409500 25136 leader_election.cc:341] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ef46f3beb7049288f0dd6d2de848a71 (127.8.102.3:46003): Not found: Tablet not found: 8f8fa211f6e0449bb8f8bf331e82ef43
I20260926 15:04:02.409516 25136 leader_election.cc:302] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9cf31f40b5b34e36b5840dead2e75f4f; no voters: 1ef46f3beb7049288f0dd6d2de848a71, b339edec4d20486393117d522b04f410
I20260926 15:04:02.409632 25539 raft_consensus.cc:2726] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:04:02.575552 25540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:02.575306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.1:43205)
0926 15:04:02.575456 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:02.575485 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":91,"threads_started":1}
I20260926 15:04:02.645501 25541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:02.645288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.2:42041)
0926 15:04:02.645418 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:02.645457 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":58,"threads_started":1}
I20260926 15:04:02.772367 25542 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:02.772109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.3:43703)
0926 15:04:02.772278 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:02.772321 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":149,"thread_start_us":80,"threads_started":1}
I20260926 15:04:03.576363 25543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:03.576124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.1:44807)
0926 15:04:03.576266 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:03.576308 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260926 15:04:03.646368 25544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:03.646100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.2:34037)
0926 15:04:03.646272 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:03.646318 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":79,"threads_started":1}
I20260926 15:04:03.773190 25545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:03.772945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.3:52781)
0926 15:04:03.773089 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:03.773138 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260926 15:04:04.577179 25546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:04.576899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.1:36475)
0926 15:04:04.577079 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:04.577128 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":66,"threads_started":1}
I20260926 15:04:04.647235 25547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:04.646999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.2:45873)
0926 15:04:04.647147 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:04.647188 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":57,"threads_started":1}
I20260926 15:04:04.774093 25548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:04.773760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.3:40713)
0926 15:04:04.774000 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:04.774038 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":182,"threads_started":1}
I20260926 15:04:05.577996 25549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:05.577745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.1:49175)
0926 15:04:05.577894 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:05.577945 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":66,"threads_started":1}
I20260926 15:04:05.648186 25550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:05.647917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.2:58851)
0926 15:04:05.648091 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:05.648137 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":74,"threads_started":1}
I20260926 15:04:05.775141 25551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:05.774784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.3:49475)
0926 15:04:05.774940 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:05.775089 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":61,"threads_started":1}
I20260926 15:04:06.185181 25552 raft_consensus.cc:482] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:04:06.185290 25552 raft_consensus.cc:504] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:04:06.185504 25552 leader_election.cc:288] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b339edec4d20486393117d522b04f410 (127.8.102.2:33963), 1ef46f3beb7049288f0dd6d2de848a71 (127.8.102.3:46003)
W20260926 15:04:06.185978 25136 leader_election.cc:341] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b339edec4d20486393117d522b04f410 (127.8.102.2:33963): Not found: Tablet not found: 8f8fa211f6e0449bb8f8bf331e82ef43
W20260926 15:04:06.186064 25136 leader_election.cc:341] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ef46f3beb7049288f0dd6d2de848a71 (127.8.102.3:46003): Not found: Tablet not found: 8f8fa211f6e0449bb8f8bf331e82ef43
I20260926 15:04:06.186081 25136 leader_election.cc:302] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9cf31f40b5b34e36b5840dead2e75f4f; no voters: 1ef46f3beb7049288f0dd6d2de848a71, b339edec4d20486393117d522b04f410
I20260926 15:04:06.186199 25552 raft_consensus.cc:2726] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:04:06.579121 25553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:06.578810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.1:55181)
0926 15:04:06.578982 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:06.579069 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":117,"threads_started":1}
I20260926 15:04:06.649022 25554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:06.648737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.2:60709)
0926 15:04:06.648884 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:06.648932 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":58,"threads_started":1}
I20260926 15:04:06.776036 25555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:06.775739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.3:55611)
0926 15:04:06.775912 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:06.775978 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":70,"threads_started":1}
I20260926 15:04:07.579859 25556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:07.579614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.1:46359)
0926 15:04:07.579766 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:07.579809 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133,"thread_start_us":70,"threads_started":1}
I20260926 15:04:07.649864 25557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:07.649562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.2:52743)
0926 15:04:07.649738 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:07.649784 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":63,"threads_started":1}
I20260926 15:04:07.776959 25558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:07.776731 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.3:40697)
0926 15:04:07.776864 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:07.776910 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":62,"threads_started":1}
I20260926 15:04:08.580693 25559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:08.580461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.1:42899)
0926 15:04:08.580604 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:08.580647 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":65,"threads_started":1}
I20260926 15:04:08.650946 25560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:08.650625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.2:32951)
0926 15:04:08.650852 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:08.650898 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":208,"thread_start_us":59,"threads_started":1}
I20260926 15:04:08.777901 25561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:08.777655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.3:39549)
0926 15:04:08.777808 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:08.777852 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":71,"threads_started":1}
I20260926 15:04:09.581779 25562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:09.581414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.1:56007)
0926 15:04:09.581625 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:09.581677 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":88,"threads_started":1}
I20260926 15:04:09.651768 25563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:09.651519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.2:59121)
0926 15:04:09.651675 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:09.651721 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":73,"threads_started":1}
I20260926 15:04:09.657032 25564 raft_consensus.cc:482] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:04:09.657124 25564 raft_consensus.cc:504] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b339edec4d20486393117d522b04f410" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 33963 } } peers { permanent_uuid: "1ef46f3beb7049288f0dd6d2de848a71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 46003 } } peers { permanent_uuid: "9cf31f40b5b34e36b5840dead2e75f4f" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 43347 } }
I20260926 15:04:09.657333 25564 leader_election.cc:288] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b339edec4d20486393117d522b04f410 (127.8.102.2:33963), 1ef46f3beb7049288f0dd6d2de848a71 (127.8.102.3:46003)
W20260926 15:04:09.657753 25136 leader_election.cc:341] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b339edec4d20486393117d522b04f410 (127.8.102.2:33963): Not found: Tablet not found: 8f8fa211f6e0449bb8f8bf331e82ef43
W20260926 15:04:09.657851 25136 leader_election.cc:341] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 1ef46f3beb7049288f0dd6d2de848a71 (127.8.102.3:46003): Not found: Tablet not found: 8f8fa211f6e0449bb8f8bf331e82ef43
I20260926 15:04:09.657888 25136 leader_election.cc:302] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 9cf31f40b5b34e36b5840dead2e75f4f; no voters: 1ef46f3beb7049288f0dd6d2de848a71, b339edec4d20486393117d522b04f410
I20260926 15:04:09.658027 25564 raft_consensus.cc:2726] T 8f8fa211f6e0449bb8f8bf331e82ef43 P 9cf31f40b5b34e36b5840dead2e75f4f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260926 15:04:09.778864 25565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:09.778551 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.3:52207)
0926 15:04:09.778768 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:09.778816 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":195,"thread_start_us":66,"threads_started":1}
I20260926 15:04:10.582777 25566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:10.582466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.1:45207)
0926 15:04:10.582668 (+ 202us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:10.582718 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":67,"threads_started":1}
I20260926 15:04:10.652616 25567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:10.652346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.2:47251)
0926 15:04:10.652508 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:10.652568 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136,"thread_start_us":64,"threads_started":1}
I20260926 15:04:10.779685 25568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:10.779407 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.3:53097)
0926 15:04:10.779581 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:10.779630 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":65,"threads_started":1}
I20260926 15:04:11.583705 25569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:11.583433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.1:46439)
0926 15:04:11.583608 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:11.583657 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":64,"threads_started":1}
I20260926 15:04:11.653385 25570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:11.653158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.2:54947)
0926 15:04:11.653298 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:11.653338 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":78,"threads_started":1}
I20260926 15:04:11.777752 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 25085
I20260926 15:04:11.780838 25571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:11.780585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36407 (local address 127.8.102.3:46071)
0926 15:04:11.780743 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:11.780790 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.62:36407: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":103,"threads_started":1}
I20260926 15:04:11.783277 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 25254
I20260926 15:04:11.787768 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 25386
2026-09-26T15:04:11Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58736 ms)
[----------] 30 tests from SecurityITest (241433 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:11 dist-test-slave-jnhs krb5kdc[25574](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:11 dist-test-slave-jnhs krb5kdc[25574](info): set up 2 sockets
Sep 26 15:04:11 dist-test-slave-jnhs krb5kdc[25574](info): commencing operation
krb5kdc: starting...
W20260926 15:04:12.890589 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.082s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:12 dist-test-slave-jnhs krb5kdc[25574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:04:13.018895 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:40665
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:40665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:13.101768 25587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:13.101984 25587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:13.102005 25587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:13.103174 25587 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:13.103215 25587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:13.103230 25587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:13.103243 25587 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:13.103255 25587 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:13.104684 25587 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:40665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:40665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:13.104894 25587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:13.105072 25587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:13.110597 25595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:13.110653 25593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:13.110736 25587 server_base.cc:1034] running on GCE node
W20260926 15:04:13.110611 25592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:13.111172 25587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:13.111546 25587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:13.112732 25587 hybrid_clock.cc:648] HybridClock initialized: now 1790435053112729 us; error 38 us; skew 500 ppm
Sep 26 15:04:13 dist-test-slave-jnhs krb5kdc[25574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435053, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:13.115478 25587 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:04:13.115998 25587 webserver.cc:460] Webserver started at http://127.8.102.62:42107/ using document root <none> and password file <none>
I20260926 15:04:13.116250 25587 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:13.116320 25587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:13.116469 25587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:13.117542 25587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "f916b521310e4ff1a1526f48ecf60255"
format_stamp: "Formatted at 2026-09-26 15:04:13 on dist-test-slave-jnhs"
server_key: "880ae37067c90e03bd89c5dae41e3028"
server_key_iv: "7c9583190e43444f6dcf7adee8f0b842"
server_key_version: "encryptionkey@0"
I20260926 15:04:13.117908 25587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "f916b521310e4ff1a1526f48ecf60255"
format_stamp: "Formatted at 2026-09-26 15:04:13 on dist-test-slave-jnhs"
server_key: "880ae37067c90e03bd89c5dae41e3028"
server_key_iv: "7c9583190e43444f6dcf7adee8f0b842"
server_key_version: "encryptionkey@0"
I20260926 15:04:13.119328 25587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:04:13.120108 25602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:13.120440 25587 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:13.120551 25587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "f916b521310e4ff1a1526f48ecf60255"
format_stamp: "Formatted at 2026-09-26 15:04:13 on dist-test-slave-jnhs"
server_key: "880ae37067c90e03bd89c5dae41e3028"
server_key_iv: "7c9583190e43444f6dcf7adee8f0b842"
server_key_version: "encryptionkey@0"
I20260926 15:04:13.120630 25587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:13.131995 25587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:13.133876 25587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:13.134044 25587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:13.138507 25587 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:40665
I20260926 15:04:13.138545 25654 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:40665 every 8 connection(s)
I20260926 15:04:13.138974 25587 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:13.139883 25655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:13.142354 25655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255: Bootstrap starting.
I20260926 15:04:13.142998 25655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:13.143316 25655 log.cc:824] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:13.143973 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 25587
I20260926 15:04:13.144080 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:13.144235 25655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255: No bootstrap required, opened a new log
I20260926 15:04:13.144279 8600 external_mini_cluster.cc:1428] Setting key a220c95a4de3242997a3eff0ce341a02
I20260926 15:04:13.146421 25655 raft_consensus.cc:348] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f916b521310e4ff1a1526f48ecf60255" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 40665 } }
I20260926 15:04:13.146617 25655 raft_consensus.cc:374] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:13.146683 25655 raft_consensus.cc:729] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f916b521310e4ff1a1526f48ecf60255, State: Initialized, Role: FOLLOWER
I20260926 15:04:13.146850 25655 consensus_queue.cc:260] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f916b521310e4ff1a1526f48ecf60255" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 40665 } }
I20260926 15:04:13.146952 25655 raft_consensus.cc:388] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:13.146991 25655 raft_consensus.cc:482] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:13.147040 25655 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:13.147792 25655 raft_consensus.cc:504] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f916b521310e4ff1a1526f48ecf60255" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 40665 } }
I20260926 15:04:13.147900 25655 leader_election.cc:302] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f916b521310e4ff1a1526f48ecf60255; no voters:
I20260926 15:04:13.148044 25655 leader_election.cc:288] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:13.148116 25660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:13.148321 25660 raft_consensus.cc:686] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255 [term 1 LEADER]: Becoming Leader. State: Replica: f916b521310e4ff1a1526f48ecf60255, State: Running, Role: LEADER
I20260926 15:04:13.148339 25655 sys_catalog.cc:564] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:13.148475 25660 consensus_queue.cc:237] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f916b521310e4ff1a1526f48ecf60255" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 40665 } }
I20260926 15:04:13.148983 25661 sys_catalog.cc:455] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f916b521310e4ff1a1526f48ecf60255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f916b521310e4ff1a1526f48ecf60255" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 40665 } } }
I20260926 15:04:13.148978 25662 sys_catalog.cc:455] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f916b521310e4ff1a1526f48ecf60255. Latest consensus state: current_term: 1 leader_uuid: "f916b521310e4ff1a1526f48ecf60255" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f916b521310e4ff1a1526f48ecf60255" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 40665 } } }
I20260926 15:04:13.149102 25661 sys_catalog.cc:458] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:13.149106 25662 sys_catalog.cc:458] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:13.149477 25669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:13.150234 25669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 15:04:13 dist-test-slave-jnhs krb5kdc[25574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:04:13.152104 25669 catalog_manager.cc:1347] Generated new cluster ID: b79cc8e718a14c0fa4c88d4bfe55d60f
I20260926 15:04:13.152976 25669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:13.156776 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:13.144847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53572 (local address 127.8.102.62:40665)
0926 15:04:13.145071 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:13.145081 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:13.145113 (+ 32us) server_negotiation.cc:407] Connection header received
0926 15:04:13.145642 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:13.145651 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:13.145718 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:13.145874 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:13.146542 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.148291 (+ 1749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.149506 (+ 1215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.150257 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.150769 (+ 512us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:13.153322 (+ 2553us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:13.153326 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:13.153327 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:13.153339 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:13.155196 (+ 1857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:13.155611 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:13.155615 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:13.155615 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:13.155707 (+ 92us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:13.155934 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:13.155935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:13.155936 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:13.156193 (+ 257us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:13.156370 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:13.156465 (+ 95us) server_negotiation.cc:299] Negotiation successful
0926 15:04:13.156609 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175,"thread_start_us":116,"threads_started":1}
I20260926 15:04:13.178783 25669 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:13.179363 25669 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:13.181924 25669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255: Generated new TSK 0
I20260926 15:04:13.182235 25669 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
I20260926 15:04:13.218317 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:40665
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:13.320142 25681 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:13.320370 25681 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:13.320398 25681 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:13.321650 25681 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:13.321699 25681 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:13.321728 25681 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:04:13.323377 25681 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:40665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25681
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:13.323642 25681 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:13.323884 25681 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:13.329205 25687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:13.329208 25689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:13.329208 25686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:13.329665 25681 server_base.cc:1034] running on GCE node
I20260926 15:04:13.329879 25681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:13.330143 25681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:13.331327 25681 hybrid_clock.cc:648] HybridClock initialized: now 1790435053331300 us; error 59 us; skew 500 ppm
Sep 26 15:04:13 dist-test-slave-jnhs krb5kdc[25574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435053, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:13.334082 25681 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:04:13.334611 25681 webserver.cc:460] Webserver started at http://127.8.102.1:38783/ using document root <none> and password file <none>
I20260926 15:04:13.334924 25681 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:13.334996 25681 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:13.335147 25681 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:13.336406 25681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "17cac225acc94ad5a23a1d138ab9cd14"
format_stamp: "Formatted at 2026-09-26 15:04:13 on dist-test-slave-jnhs"
server_key: "dd658a73b6dcc442f36dfed14814f253"
server_key_iv: "a50b3950f86e05a85318ae5505150780"
server_key_version: "encryptionkey@0"
I20260926 15:04:13.336889 25681 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "17cac225acc94ad5a23a1d138ab9cd14"
format_stamp: "Formatted at 2026-09-26 15:04:13 on dist-test-slave-jnhs"
server_key: "dd658a73b6dcc442f36dfed14814f253"
server_key_iv: "a50b3950f86e05a85318ae5505150780"
server_key_version: "encryptionkey@0"
I20260926 15:04:13.338402 25681 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:04:13.339330 25696 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:13.339613 25681 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:04:13.339717 25681 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "17cac225acc94ad5a23a1d138ab9cd14"
format_stamp: "Formatted at 2026-09-26 15:04:13 on dist-test-slave-jnhs"
server_key: "dd658a73b6dcc442f36dfed14814f253"
server_key_iv: "a50b3950f86e05a85318ae5505150780"
server_key_version: "encryptionkey@0"
I20260926 15:04:13.339797 25681 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:13.356243 25681 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:13.358692 25681 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:13.358840 25681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:13.359125 25681 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:04:13.359457 25681 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:04:13.359512 25681 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:13.359555 25681 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:04:13.359582 25681 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:13.365727 25681 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:36193
I20260926 15:04:13.365945 25809 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:36193 every 8 connection(s)
I20260926 15:04:13.366209 25681 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
Sep 26 15:04:13 dist-test-slave-jnhs krb5kdc[25574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435053, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:04:13.374593 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 25681
I20260926 15:04:13.374742 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:04:13.374976 8600 external_mini_cluster.cc:1428] Setting key f74fa0599cf6ee68d947d4fb623ed879
I20260926 15:04:13.376183 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:13.367065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:52339 (local address 127.8.102.62:40665)
0926 15:04:13.367148 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:13.367150 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:13.367303 (+ 153us) server_negotiation.cc:407] Connection header received
0926 15:04:13.367851 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:13.367852 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:13.367863 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:13.367935 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:13.368880 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.369399 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.370412 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.370664 (+ 252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.370723 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:13.373096 (+ 2373us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:13.373098 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:13.373098 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:13.373105 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:13.374390 (+ 1285us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:13.374974 (+ 584us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:13.374975 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:13.374975 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:13.375056 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:13.375358 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:13.375359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:13.375360 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:13.375509 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:13.375579 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:13.375972 (+ 393us) server_negotiation.cc:299] Negotiation successful
0926 15:04:13.376073 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 15:04:13.376356 25812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:13.366986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:40665 (local address 127.8.102.1:52339)
0926 15:04:13.367168 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:13.367211 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:13.367732 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:13.367990 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:13.367994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:13.368563 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:13.368771 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:13.368777 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.369496 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.369497 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:13.370318 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:13.370320 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.370733 (+ 413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.370734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:13.370839 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:13.371429 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:13.371438 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:13.372988 (+ 1550us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:13.374496 (+ 1508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:13.374504 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:13.374507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:13.374897 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:13.375151 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:13.375152 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:13.375153 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:13.375281 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:13.375598 (+ 317us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:13.375605 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:13.375816 (+ 211us) client_negotiation.cc:769] Sending connection context
0926 15:04:13.375931 (+ 115us) client_negotiation.cc:240] Negotiation successful
0926 15:04:13.376140 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"thread_start_us":89,"threads_started":1}
I20260926 15:04:13.377652 25810 heartbeater.cc:347] Connected to a master server at 127.8.102.62:40665
I20260926 15:04:13.377779 25810 heartbeater.cc:464] Registering TS with master...
I20260926 15:04:13.377969 25810 heartbeater.cc:511] Master 127.8.102.62:40665 requested a full tablet report, sending...
I20260926 15:04:13.378531 25619 ts_manager.cc:194] Registered new tserver with Master: 17cac225acc94ad5a23a1d138ab9cd14 (127.8.102.1:36193)
I20260926 15:04:13.379208 25619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:52339
WARNING: no policy specified for kudu/127.8.102.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
I20260926 15:04:13.402060 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:40665
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:13.485635 25815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:13.485864 25815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:13.485890 25815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:13.487176 25815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:13.487226 25815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:13.487253 25815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:04:13.488837 25815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:40665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:13.489109 25815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:13.489356 25815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:13.494715 25823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:13.494717 25820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:13.494709 25821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:13.495338 25815 server_base.cc:1034] running on GCE node
I20260926 15:04:13.495602 25815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:13.495904 25815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:13.497143 25815 hybrid_clock.cc:648] HybridClock initialized: now 1790435053497106 us; error 39 us; skew 500 ppm
Sep 26 15:04:13 dist-test-slave-jnhs krb5kdc[25574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435053, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:13.499851 25815 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:04:13.500396 25815 webserver.cc:460] Webserver started at http://127.8.102.2:35435/ using document root <none> and password file <none>
I20260926 15:04:13.500610 25815 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:13.500653 25815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:13.500746 25815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:13.501767 25815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "21642652bbe74cc48fa64ff426c0c02f"
format_stamp: "Formatted at 2026-09-26 15:04:13 on dist-test-slave-jnhs"
server_key: "e6f469237a40c8089c8da25660677268"
server_key_iv: "3476667c77ca834ad8a5fbd41315e4c2"
server_key_version: "encryptionkey@0"
I20260926 15:04:13.502084 25815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "21642652bbe74cc48fa64ff426c0c02f"
format_stamp: "Formatted at 2026-09-26 15:04:13 on dist-test-slave-jnhs"
server_key: "e6f469237a40c8089c8da25660677268"
server_key_iv: "3476667c77ca834ad8a5fbd41315e4c2"
server_key_version: "encryptionkey@0"
I20260926 15:04:13.503533 25815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 15:04:13.504381 25830 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:13.504642 25815 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:04:13.504726 25815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "21642652bbe74cc48fa64ff426c0c02f"
format_stamp: "Formatted at 2026-09-26 15:04:13 on dist-test-slave-jnhs"
server_key: "e6f469237a40c8089c8da25660677268"
server_key_iv: "3476667c77ca834ad8a5fbd41315e4c2"
server_key_version: "encryptionkey@0"
I20260926 15:04:13.504786 25815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:13.521788 25815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:13.524739 25815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:13.524868 25815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:13.525079 25815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:04:13.525346 25815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:04:13.525380 25815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:13.525460 25815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:04:13.525496 25815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:13.531920 25815 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:37471
I20260926 15:04:13.532009 25943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:37471 every 8 connection(s)
I20260926 15:04:13.532372 25815 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:04:13.537385 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 25815
I20260926 15:04:13.537477 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:04:13.537721 8600 external_mini_cluster.cc:1428] Setting key ccde4309506ae222b6a7887c4a4d5842
Sep 26 15:04:13 dist-test-slave-jnhs krb5kdc[25574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435053, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:04:13.543618 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:13.533221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:38661 (local address 127.8.102.62:40665)
0926 15:04:13.533304 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:13.533307 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:13.533453 (+ 146us) server_negotiation.cc:407] Connection header received
0926 15:04:13.533970 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:13.533972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:13.533984 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:13.534046 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:13.535245 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.535771 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.536952 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.537187 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.537250 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:13.540470 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:13.540472 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:13.540472 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:13.540480 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:13.542059 (+ 1579us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:13.542540 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:13.542541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:13.542541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:13.542658 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:13.542917 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:13.542918 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:13.542918 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:13.543051 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:13.543129 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:13.543434 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 15:04:13.543523 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260926 15:04:13.543872 25946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:13.533120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:40665 (local address 127.8.102.2:38661)
0926 15:04:13.533301 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:13.533343 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:13.533857 (+ 514us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:13.534105 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:13.534109 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:13.534810 (+ 701us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:13.535136 (+ 326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:13.535143 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.535872 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.535873 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:13.536855 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:13.536858 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.537640 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.537641 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:13.537814 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:13.538497 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:13.538509 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:13.540344 (+ 1835us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:13.542186 (+ 1842us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:13.542194 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:13.542197 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:13.542434 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:13.542718 (+ 284us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:13.542719 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:13.542720 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:13.542840 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:13.543132 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:13.543139 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:13.543290 (+ 151us) client_negotiation.cc:769] Sending connection context
0926 15:04:13.543398 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 15:04:13.543581 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":67,"threads_started":1}
I20260926 15:04:13.544312 25944 heartbeater.cc:347] Connected to a master server at 127.8.102.62:40665
I20260926 15:04:13.544401 25944 heartbeater.cc:464] Registering TS with master...
I20260926 15:04:13.544610 25944 heartbeater.cc:511] Master 127.8.102.62:40665 requested a full tablet report, sending...
I20260926 15:04:13.545032 25619 ts_manager.cc:194] Registered new tserver with Master: 21642652bbe74cc48fa64ff426c0c02f (127.8.102.2:37471)
I20260926 15:04:13.545430 25619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:38661
WARNING: no policy specified for kudu/127.8.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
I20260926 15:04:13.566864 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:40665
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:36633
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:13.652469 25949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:13.652717 25949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:13.652745 25949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:13.653995 25949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:13.654045 25949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:13.654075 25949 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:04:13.655680 25949 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36633
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:40665
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25949
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:13.655943 25949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:13.656207 25949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:13.661659 25954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:13.661669 25957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:13.661664 25955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:13.661703 25949 server_base.cc:1034] running on GCE node
I20260926 15:04:13.662051 25949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:13.662281 25949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:13.663447 25949 hybrid_clock.cc:648] HybridClock initialized: now 1790435053663412 us; error 60 us; skew 500 ppm
Sep 26 15:04:13 dist-test-slave-jnhs krb5kdc[25574](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435053, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:13.666270 25949 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:04:13.666877 25949 webserver.cc:460] Webserver started at http://127.8.102.3:36187/ using document root <none> and password file <none>
I20260926 15:04:13.667158 25949 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:13.667232 25949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:13.667384 25949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:13.668474 25949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "92b48d1b65044db2afe57837ead2bc71"
format_stamp: "Formatted at 2026-09-26 15:04:13 on dist-test-slave-jnhs"
server_key: "ecdf3711275a2aabae4f5eebd9b7b148"
server_key_iv: "4a197ee9800c96755df8c9d4379525c3"
server_key_version: "encryptionkey@0"
I20260926 15:04:13.668884 25949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "92b48d1b65044db2afe57837ead2bc71"
format_stamp: "Formatted at 2026-09-26 15:04:13 on dist-test-slave-jnhs"
server_key: "ecdf3711275a2aabae4f5eebd9b7b148"
server_key_iv: "4a197ee9800c96755df8c9d4379525c3"
server_key_version: "encryptionkey@0"
I20260926 15:04:13.670296 25949 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:04:13.671248 25964 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:13.671500 25949 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 15:04:13.671674 25949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "92b48d1b65044db2afe57837ead2bc71"
format_stamp: "Formatted at 2026-09-26 15:04:13 on dist-test-slave-jnhs"
server_key: "ecdf3711275a2aabae4f5eebd9b7b148"
server_key_iv: "4a197ee9800c96755df8c9d4379525c3"
server_key_version: "encryptionkey@0"
I20260926 15:04:13.671758 25949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:13.680935 25949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:13.683580 25949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:13.683775 25949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:13.684036 25949 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:04:13.684387 25949 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:04:13.684448 25949 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:13.684497 25949 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:04:13.684525 25949 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:13.691147 25949 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:44701
I20260926 15:04:13.691195 26077 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:44701 every 8 connection(s)
I20260926 15:04:13.691560 25949 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:04:13.692431 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 25949
I20260926 15:04:13.692548 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:04:13.692768 8600 external_mini_cluster.cc:1428] Setting key c6f51d3b0d700081846574c1f39d9b62
Sep 26 15:04:13 dist-test-slave-jnhs krb5kdc[25574](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435053, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:04:13.702375 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:13.692528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:56087 (local address 127.8.102.62:40665)
0926 15:04:13.692640 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:13.692643 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:13.692934 (+ 291us) server_negotiation.cc:407] Connection header received
0926 15:04:13.693703 (+ 769us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:13.693704 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:13.693715 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:13.693786 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:13.695104 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.695660 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.696791 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.697009 (+ 218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.697117 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:13.699504 (+ 2387us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:13.699505 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:13.699506 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:13.699513 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:13.701081 (+ 1568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:13.701508 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:13.701509 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:13.701509 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:13.701559 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:13.701795 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:13.701795 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:13.701796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:13.701899 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:13.701948 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:13.702236 (+ 288us) server_negotiation.cc:299] Negotiation successful
0926 15:04:13.702301 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260926 15:04:13.702608 26080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:13.692385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:40665 (local address 127.8.102.3:56087)
0926 15:04:13.692740 (+ 355us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:13.692794 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:13.693599 (+ 805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:13.693909 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:13.693914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:13.694613 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:13.694923 (+ 310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:13.694930 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.695777 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.695778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:13.696695 (+ 917us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:13.696697 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.697133 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.697133 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:13.697245 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:13.697827 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:13.697836 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:13.699393 (+ 1557us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:13.701173 (+ 1780us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:13.701180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:13.701182 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:13.701439 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:13.701626 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:13.701627 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:13.701627 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:13.701738 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:13.701962 (+ 224us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:13.701967 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:13.702111 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 15:04:13.702211 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 15:04:13.702349 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":304,"thread_start_us":90,"threads_started":1}
I20260926 15:04:13.703191 26078 heartbeater.cc:347] Connected to a master server at 127.8.102.62:40665
I20260926 15:04:13.703295 26078 heartbeater.cc:464] Registering TS with master...
I20260926 15:04:13.703454 26078 heartbeater.cc:511] Master 127.8.102.62:40665 requested a full tablet report, sending...
I20260926 15:04:13.703861 25619 ts_manager.cc:194] Registered new tserver with Master: 92b48d1b65044db2afe57837ead2bc71 (127.8.102.3:44701)
I20260926 15:04:13.704311 25619 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:56087
I20260926 15:04:13.705016 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:04:13.713316 25659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:13.706491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53588 (local address 127.8.102.62:40665)
0926 15:04:13.706570 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:13.706572 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:13.706697 (+ 125us) server_negotiation.cc:407] Connection header received
0926 15:04:13.706790 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:13.706791 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:13.706801 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:13.706846 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:13.707484 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.708046 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.708811 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.709051 (+ 240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.709134 (+ 83us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:13.710446 (+ 1312us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:13.710448 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:13.710448 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:13.710455 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:13.711960 (+ 1505us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:13.712393 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:13.712394 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:13.712395 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:13.712460 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:13.712683 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:13.712685 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:13.712685 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:13.712853 (+ 168us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:13.712972 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:13.713102 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 15:04:13.713199 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 15:04:13.715327 25619 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:04:13.717046 25619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:04:13.724859 26090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:13.719846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:37471 (local address 127.0.0.1:56900)
0926 15:04:13.720016 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:13.720062 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:13.720148 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:13.720578 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:13.720582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:13.720584 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:13.720668 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:13.720670 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.722176 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.722177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:13.723541 (+ 1364us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:13.723544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.724524 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.724525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:13.724600 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:13.724612 (+ 12us) client_negotiation.cc:769] Sending connection context
0926 15:04:13.724676 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 15:04:13.724751 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"mutex_wait_us":3,"thread_start_us":79,"threads_started":1}
I20260926 15:04:13.724973 26089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:13.719855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:36193 (local address 127.0.0.1:54526)
0926 15:04:13.720224 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:13.720238 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:13.720314 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:13.720623 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:13.720624 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:13.720626 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:13.720693 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:13.720695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.721967 (+ 1272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.721968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:13.723725 (+ 1757us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:13.723726 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.724677 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.724678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:13.724740 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:13.724743 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:04:13.724808 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 15:04:13.724844 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":355,"mutex_wait_us":39,"thread_start_us":103,"threads_started":1}
I20260926 15:04:13.724977 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:13.720063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54526 (local address 127.8.102.1:36193)
0926 15:04:13.720232 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:13.720239 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:13.720308 (+ 69us) server_negotiation.cc:407] Connection header received
0926 15:04:13.720395 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:13.720398 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:13.720415 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:13.720510 (+ 95us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:13.720786 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.721841 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.723851 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.724575 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.724671 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:13.724738 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:13.724822 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 15:04:13.724872 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":75,"threads_started":1}
I20260926 15:04:13.725348 26091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:13.719954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56900 (local address 127.8.102.2:37471)
0926 15:04:13.720323 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:13.720328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:13.720338 (+ 10us) server_negotiation.cc:407] Connection header received
0926 15:04:13.720383 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:13.720386 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:13.720402 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:13.720571 (+ 169us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:13.720767 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.722064 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.723635 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.724393 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.725102 (+ 709us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:13.725181 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:13.725205 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 15:04:13.725240 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":330,"thread_start_us":97,"threads_started":1}
I20260926 15:04:13.725456 26094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:13.720290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54060 (local address 127.8.102.3:44701)
0926 15:04:13.720502 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:13.720508 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:13.720808 (+ 300us) server_negotiation.cc:407] Connection header received
0926 15:04:13.720861 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:13.720864 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:13.720880 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:13.720972 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:13.721101 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.722283 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.723552 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.724405 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.724478 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:13.724548 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:13.724732 (+ 184us) server_negotiation.cc:299] Negotiation successful
0926 15:04:13.725361 (+ 629us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":178,"thread_start_us":77,"threads_started":1}
I20260926 15:04:13.725575 26093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:13.720090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:44701 (local address 127.0.0.1:54060)
0926 15:04:13.720674 (+ 584us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:13.720685 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:13.720739 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:13.720979 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:13.720980 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:13.720982 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:13.721031 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:13.721032 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.722391 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.722392 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:13.723454 (+ 1062us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:13.723457 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.724480 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.724482 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:13.724579 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:13.724598 (+ 19us) client_negotiation.cc:769] Sending connection context
0926 15:04:13.724674 (+ 76us) client_negotiation.cc:240] Negotiation successful
0926 15:04:13.725490 (+ 816us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"mutex_wait_us":750,"thread_start_us":80,"threads_started":1}
I20260926 15:04:13.726127 25744 tablet_service.cc:1467] Processing CreateTablet for tablet 9f522f0e9e484cf3a6846b41b68ed77e (DEFAULT_TABLE table=test-table [id=a8a26120d1e64eafae81901ed95425f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:04:13.726289 26012 tablet_service.cc:1467] Processing CreateTablet for tablet 9f522f0e9e484cf3a6846b41b68ed77e (DEFAULT_TABLE table=test-table [id=a8a26120d1e64eafae81901ed95425f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:04:13.726426 25744 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f522f0e9e484cf3a6846b41b68ed77e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:13.726717 26012 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f522f0e9e484cf3a6846b41b68ed77e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:13.729457 26095 tablet_bootstrap.cc:492] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14: Bootstrap starting.
I20260926 15:04:13.729532 26096 tablet_bootstrap.cc:492] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71: Bootstrap starting.
I20260926 15:04:13.730171 26096 tablet_bootstrap.cc:654] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:13.730044 25878 tablet_service.cc:1467] Processing CreateTablet for tablet 9f522f0e9e484cf3a6846b41b68ed77e (DEFAULT_TABLE table=test-table [id=a8a26120d1e64eafae81901ed95425f4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:04:13.730213 26095 tablet_bootstrap.cc:654] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:13.730443 25878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f522f0e9e484cf3a6846b41b68ed77e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:13.730870 26096 log.cc:824] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:13.730898 26095 log.cc:824] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:13.731782 26095 tablet_bootstrap.cc:492] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14: No bootstrap required, opened a new log
I20260926 15:04:13.731871 26095 ts_tablet_manager.cc:1397] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 15:04:13.734197 26099 tablet_bootstrap.cc:492] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f: Bootstrap starting.
I20260926 15:04:13.734174 26095 raft_consensus.cc:348] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21642652bbe74cc48fa64ff426c0c02f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37471 } } peers { permanent_uuid: "17cac225acc94ad5a23a1d138ab9cd14" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36193 } } peers { permanent_uuid: "92b48d1b65044db2afe57837ead2bc71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44701 } }
I20260926 15:04:13.734400 26095 raft_consensus.cc:374] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:13.734457 26095 raft_consensus.cc:729] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 17cac225acc94ad5a23a1d138ab9cd14, State: Initialized, Role: FOLLOWER
I20260926 15:04:13.734619 26095 consensus_queue.cc:260] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21642652bbe74cc48fa64ff426c0c02f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37471 } } peers { permanent_uuid: "17cac225acc94ad5a23a1d138ab9cd14" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36193 } } peers { permanent_uuid: "92b48d1b65044db2afe57837ead2bc71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44701 } }
I20260926 15:04:13.734980 26095 ts_tablet_manager.cc:1428] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260926 15:04:13.734982 26099 tablet_bootstrap.cc:654] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:13.735033 26096 tablet_bootstrap.cc:492] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71: No bootstrap required, opened a new log
I20260926 15:04:13.735095 26096 ts_tablet_manager.cc:1397] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.000s
I20260926 15:04:13.735128 25810 heartbeater.cc:503] Master 127.8.102.62:40665 was elected leader, sending a full tablet report...
I20260926 15:04:13.735761 26099 log.cc:824] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:13.736537 26099 tablet_bootstrap.cc:492] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f: No bootstrap required, opened a new log
I20260926 15:04:13.736605 26099 ts_tablet_manager.cc:1397] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260926 15:04:13.737345 26096 raft_consensus.cc:348] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21642652bbe74cc48fa64ff426c0c02f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37471 } } peers { permanent_uuid: "17cac225acc94ad5a23a1d138ab9cd14" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36193 } } peers { permanent_uuid: "92b48d1b65044db2afe57837ead2bc71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44701 } }
I20260926 15:04:13.737517 26096 raft_consensus.cc:374] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:13.737566 26096 raft_consensus.cc:729] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 92b48d1b65044db2afe57837ead2bc71, State: Initialized, Role: FOLLOWER
I20260926 15:04:13.737708 26096 consensus_queue.cc:260] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21642652bbe74cc48fa64ff426c0c02f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37471 } } peers { permanent_uuid: "17cac225acc94ad5a23a1d138ab9cd14" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36193 } } peers { permanent_uuid: "92b48d1b65044db2afe57837ead2bc71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44701 } }
I20260926 15:04:13.738047 26096 ts_tablet_manager.cc:1428] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 15:04:13.738078 26078 heartbeater.cc:503] Master 127.8.102.62:40665 was elected leader, sending a full tablet report...
I20260926 15:04:13.738711 26099 raft_consensus.cc:348] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21642652bbe74cc48fa64ff426c0c02f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37471 } } peers { permanent_uuid: "17cac225acc94ad5a23a1d138ab9cd14" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36193 } } peers { permanent_uuid: "92b48d1b65044db2afe57837ead2bc71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44701 } }
I20260926 15:04:13.738901 26099 raft_consensus.cc:374] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:13.738976 26099 raft_consensus.cc:729] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21642652bbe74cc48fa64ff426c0c02f, State: Initialized, Role: FOLLOWER
I20260926 15:04:13.739105 26099 consensus_queue.cc:260] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21642652bbe74cc48fa64ff426c0c02f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37471 } } peers { permanent_uuid: "17cac225acc94ad5a23a1d138ab9cd14" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36193 } } peers { permanent_uuid: "92b48d1b65044db2afe57837ead2bc71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44701 } }
I20260926 15:04:13.739384 26099 ts_tablet_manager.cc:1428] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 15:04:13.739529 25944 heartbeater.cc:503] Master 127.8.102.62:40665 was elected leader, sending a full tablet report...
W20260926 15:04:13.782992 25945 tablet.cc:2369] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:04:13.792237 26100 raft_consensus.cc:482] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:04:13.792373 26100 raft_consensus.cc:504] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21642652bbe74cc48fa64ff426c0c02f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37471 } } peers { permanent_uuid: "17cac225acc94ad5a23a1d138ab9cd14" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36193 } } peers { permanent_uuid: "92b48d1b65044db2afe57837ead2bc71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44701 } }
I20260926 15:04:13.792896 26100 leader_election.cc:288] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 21642652bbe74cc48fa64ff426c0c02f (127.8.102.2:37471), 92b48d1b65044db2afe57837ead2bc71 (127.8.102.3:44701)
I20260926 15:04:13.796327 25812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:13.793209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:37471 (local address 127.8.102.1:49313)
0926 15:04:13.793276 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:13.793288 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:13.793360 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:13.793657 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:13.793658 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:13.793660 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:13.793734 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:13.793736 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.794539 (+ 803us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.794540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:13.795350 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:13.795352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.796054 (+ 702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.796056 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:13.796146 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:13.796150 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:04:13.796207 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 15:04:13.796256 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 15:04:13.796363 26091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:13.793234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:49313 (local address 127.8.102.2:37471)
0926 15:04:13.793291 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:13.793294 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:13.793445 (+ 151us) server_negotiation.cc:407] Connection header received
0926 15:04:13.793479 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:13.793480 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:13.793493 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:13.793598 (+ 105us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:13.793822 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.794422 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.795452 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.795946 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.796019 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:13.796037 (+ 18us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:13.796233 (+ 196us) server_negotiation.cc:299] Negotiation successful
0926 15:04:13.796275 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260926 15:04:13.796532 26094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:13.793301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:58695 (local address 127.8.102.3:44701)
0926 15:04:13.793360 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:13.793362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:13.793572 (+ 210us) server_negotiation.cc:407] Connection header received
0926 15:04:13.793633 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:13.793635 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:13.793648 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:13.793703 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:13.793977 (+ 274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.794506 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.795638 (+ 1132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.796125 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.796200 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:13.796212 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:13.796405 (+ 193us) server_negotiation.cc:299] Negotiation successful
0926 15:04:13.796450 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 15:04:13.796546 26104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:13.793222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:44701 (local address 127.8.102.1:58695)
0926 15:04:13.793442 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:13.793476 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:13.793563 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:13.793824 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:13.793824 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:13.793826 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:13.793902 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:13.793904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.794598 (+ 694us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.794599 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:13.795545 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:13.795548 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.796238 (+ 690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.796239 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:13.796318 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:13.796322 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:04:13.796385 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 15:04:13.796431 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":194,"thread_start_us":123,"threads_started":1}
I20260926 15:04:13.796756 25898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f522f0e9e484cf3a6846b41b68ed77e" candidate_uuid: "17cac225acc94ad5a23a1d138ab9cd14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21642652bbe74cc48fa64ff426c0c02f" is_pre_election: true
I20260926 15:04:13.796924 25898 raft_consensus.cc:2454] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17cac225acc94ad5a23a1d138ab9cd14 in term 0.
I20260926 15:04:13.796936 26032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f522f0e9e484cf3a6846b41b68ed77e" candidate_uuid: "17cac225acc94ad5a23a1d138ab9cd14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b48d1b65044db2afe57837ead2bc71" is_pre_election: true
I20260926 15:04:13.797133 26032 raft_consensus.cc:2454] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 17cac225acc94ad5a23a1d138ab9cd14 in term 0.
I20260926 15:04:13.797257 25698 leader_election.cc:302] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17cac225acc94ad5a23a1d138ab9cd14, 21642652bbe74cc48fa64ff426c0c02f; no voters:
I20260926 15:04:13.797425 26100 raft_consensus.cc:2781] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:04:13.797493 26100 raft_consensus.cc:482] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:04:13.797511 26100 raft_consensus.cc:3037] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:13.798231 26100 raft_consensus.cc:504] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21642652bbe74cc48fa64ff426c0c02f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37471 } } peers { permanent_uuid: "17cac225acc94ad5a23a1d138ab9cd14" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36193 } } peers { permanent_uuid: "92b48d1b65044db2afe57837ead2bc71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44701 } }
I20260926 15:04:13.798384 26100 leader_election.cc:288] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14 [CANDIDATE]: Term 1 election: Requested vote from peers 21642652bbe74cc48fa64ff426c0c02f (127.8.102.2:37471), 92b48d1b65044db2afe57837ead2bc71 (127.8.102.3:44701)
I20260926 15:04:13.798568 25898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f522f0e9e484cf3a6846b41b68ed77e" candidate_uuid: "17cac225acc94ad5a23a1d138ab9cd14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21642652bbe74cc48fa64ff426c0c02f"
I20260926 15:04:13.798555 26032 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f522f0e9e484cf3a6846b41b68ed77e" candidate_uuid: "17cac225acc94ad5a23a1d138ab9cd14" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "92b48d1b65044db2afe57837ead2bc71"
I20260926 15:04:13.798700 25898 raft_consensus.cc:3037] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:13.798702 26032 raft_consensus.cc:3037] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:13.799517 26032 raft_consensus.cc:2454] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17cac225acc94ad5a23a1d138ab9cd14 in term 1.
I20260926 15:04:13.799543 25898 raft_consensus.cc:2454] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 17cac225acc94ad5a23a1d138ab9cd14 in term 1.
I20260926 15:04:13.799728 25700 leader_election.cc:302] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 17cac225acc94ad5a23a1d138ab9cd14, 92b48d1b65044db2afe57837ead2bc71; no voters:
I20260926 15:04:13.799829 26100 raft_consensus.cc:2781] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:13.800004 26100 raft_consensus.cc:686] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14 [term 1 LEADER]: Becoming Leader. State: Replica: 17cac225acc94ad5a23a1d138ab9cd14, State: Running, Role: LEADER
I20260926 15:04:13.800117 26100 consensus_queue.cc:237] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21642652bbe74cc48fa64ff426c0c02f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37471 } } peers { permanent_uuid: "17cac225acc94ad5a23a1d138ab9cd14" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36193 } } peers { permanent_uuid: "92b48d1b65044db2afe57837ead2bc71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44701 } }
I20260926 15:04:13.801087 25619 catalog_manager.cc:5495] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14 reported cstate change: term changed from 0 to 1, leader changed from <none> to 17cac225acc94ad5a23a1d138ab9cd14 (127.8.102.1). New cstate: current_term: 1 leader_uuid: "17cac225acc94ad5a23a1d138ab9cd14" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "21642652bbe74cc48fa64ff426c0c02f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37471 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "17cac225acc94ad5a23a1d138ab9cd14" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36193 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "92b48d1b65044db2afe57837ead2bc71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44701 } health_report { overall_health: UNKNOWN } } }
I20260926 15:04:13.817056 26092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:13.814532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54534 (local address 127.8.102.1:36193)
0926 15:04:13.814601 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:13.814603 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:13.814612 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:04:13.814682 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:13.814683 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:13.814701 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:13.814779 (+ 78us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:04:13.814916 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.815429 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.816234 (+ 805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:13.816458 (+ 224us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:13.816535 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:13.816708 (+ 173us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:04:13.816820 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:04:13.816917 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:13.816937 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 15:04:13.816957 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 15:04:13.819082 25898 raft_consensus.cc:1261] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f [term 1 FOLLOWER]: Refusing update from remote peer 17cac225acc94ad5a23a1d138ab9cd14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:04:13.819082 26032 raft_consensus.cc:1261] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71 [term 1 FOLLOWER]: Refusing update from remote peer 17cac225acc94ad5a23a1d138ab9cd14: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:04:13.819521 26105 consensus_queue.cc:1035] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "92b48d1b65044db2afe57837ead2bc71" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44701 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:04:13.819684 26100 consensus_queue.cc:1035] T 9f522f0e9e484cf3a6846b41b68ed77e P 17cac225acc94ad5a23a1d138ab9cd14 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21642652bbe74cc48fa64ff426c0c02f" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 37471 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:04:13.822068 26109 mvcc.cc:205] Tried to move back new op lower bound from 7333621980439531520 to 7333621980365815808. Current Snapshot: MvccSnapshot[applied={T|T < 7333621980439531520}]
I20260926 15:04:13.822937 26110 mvcc.cc:205] Tried to move back new op lower bound from 7333621980439531520 to 7333621980365815808. Current Snapshot: MvccSnapshot[applied={T|T < 7333621980439531520}]
I20260926 15:04:13.824913 25619 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:04:13.825017 25619 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:04:13.825810 25619 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f916b521310e4ff1a1526f48ecf60255: Sending DeleteTablet for 3 replicas of tablet 9f522f0e9e484cf3a6846b41b68ed77e
I20260926 15:04:13.826143 25878 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f522f0e9e484cf3a6846b41b68ed77e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:04:13 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:56900
I20260926 15:04:13.826182 26012 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f522f0e9e484cf3a6846b41b68ed77e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:04:13 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:54060
I20260926 15:04:13.826473 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 25681
I20260926 15:04:13.826496 25744 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f522f0e9e484cf3a6846b41b68ed77e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:04:13 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:54526
I20260926 15:04:13.827095 26118 tablet_replica.cc:330] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f: stopping tablet replica
I20260926 15:04:13.827196 26118 raft_consensus.cc:2229] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:04:13.827286 26118 raft_consensus.cc:2258] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:04:13.827442 26119 tablet_replica.cc:330] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71: stopping tablet replica
I20260926 15:04:13.827555 26119 raft_consensus.cc:2229] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:04:13.827661 26119 raft_consensus.cc:2258] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:04:13.828148 26119 ts_tablet_manager.cc:1905] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:04:13.830663 26119 ts_tablet_manager.cc:1918] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:04:13.830747 26119 log.cc:1196] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/9f522f0e9e484cf3a6846b41b68ed77e
I20260926 15:04:13.830803 26118 ts_tablet_manager.cc:1905] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:04:13.831061 26119 ts_tablet_manager.cc:1939] T 9f522f0e9e484cf3a6846b41b68ed77e P 92b48d1b65044db2afe57837ead2bc71: Deleting consensus metadata
I20260926 15:04:13.831873 25606 catalog_manager.cc:4841] TS 92b48d1b65044db2afe57837ead2bc71 (127.8.102.3:44701): tablet 9f522f0e9e484cf3a6846b41b68ed77e (table test-table [id=a8a26120d1e64eafae81901ed95425f4]) successfully deleted
I20260926 15:04:13.832356 26118 ts_tablet_manager.cc:1918] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:04:13.832410 26118 log.cc:1196] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/9f522f0e9e484cf3a6846b41b68ed77e
I20260926 15:04:13.832620 26118 ts_tablet_manager.cc:1939] T 9f522f0e9e484cf3a6846b41b68ed77e P 21642652bbe74cc48fa64ff426c0c02f: Deleting consensus metadata
I20260926 15:04:13.833040 25604 catalog_manager.cc:4841] TS 21642652bbe74cc48fa64ff426c0c02f (127.8.102.2:37471): tablet 9f522f0e9e484cf3a6846b41b68ed77e (table test-table [id=a8a26120d1e64eafae81901ed95425f4]) successfully deleted
W20260926 15:04:13.834350 25606 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.102.1:36193 (error 108)
I20260926 15:04:13.834825 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 25815
I20260926 15:04:13.835536 26093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:13.834915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:36193 (local address 127.0.0.1:54540)
0926 15:04:13.835440 (+ 525us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:13.835489 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.1:36193: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500}
W20260926 15:04:13.835692 25606 catalog_manager.cc:4584] TS 17cac225acc94ad5a23a1d138ab9cd14 (127.8.102.1:36193): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f522f0e9e484cf3a6846b41b68ed77e: Network error: Client connection negotiation failed: client connection to 127.8.102.1:36193: connect: Connection refused (error 111)
I20260926 15:04:13.841017 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 25949
I20260926 15:04:13.845898 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 25587
2026-09-26T15:04:13Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (2058 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-26T15:04:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:13Z Disabled control of system clock
I20260926 15:04:13.875175 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:37057
--webserver_interface=127.8.102.62
--webserver_port=0
--builtin_ntp_servers=127.8.102.20:36355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:37057
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:04:13.958557 26124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:13.958818 26124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:13.958839 26124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:13.960058 26124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:13.960111 26124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:13.960125 26124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:13.960139 26124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:13.960150 26124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:13.961447 26124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:37057
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:37057
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26124
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:13.961666 26124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:13.961906 26124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:13.967356 26132 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:13.967362 26130 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:13.967362 26129 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:13.967849 26124 server_base.cc:1034] running on GCE node
I20260926 15:04:13.968164 26124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:13.968536 26124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:13.969735 26124 hybrid_clock.cc:648] HybridClock initialized: now 1790435053969717 us; error 53 us; skew 500 ppm
I20260926 15:04:13.971275 26124 webserver.cc:460] Webserver started at http://127.8.102.62:43097/ using document root <none> and password file <none>
I20260926 15:04:13.971541 26124 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:13.971628 26124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:13.971801 26124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:13.972916 26124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "b730f32f73a34733aaafebe90203df66"
format_stamp: "Formatted at 2026-09-26 15:04:13 on dist-test-slave-jnhs"
server_key: "163133abe75526eb6bbff117a6487113"
server_key_iv: "6dbe9c8092dce7d277829152a790b04b"
server_key_version: "encryptionkey@0"
I20260926 15:04:13.973301 26124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "b730f32f73a34733aaafebe90203df66"
format_stamp: "Formatted at 2026-09-26 15:04:13 on dist-test-slave-jnhs"
server_key: "163133abe75526eb6bbff117a6487113"
server_key_iv: "6dbe9c8092dce7d277829152a790b04b"
server_key_version: "encryptionkey@0"
I20260926 15:04:13.974689 26124 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 15:04:13.975483 26138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:13.975672 26124 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260926 15:04:13.975772 26124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "b730f32f73a34733aaafebe90203df66"
format_stamp: "Formatted at 2026-09-26 15:04:13 on dist-test-slave-jnhs"
server_key: "163133abe75526eb6bbff117a6487113"
server_key_iv: "6dbe9c8092dce7d277829152a790b04b"
server_key_version: "encryptionkey@0"
I20260926 15:04:13.975863 26124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:13.988300 26124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:13.988662 26124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:13.988821 26124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:13.993074 26124 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:37057
I20260926 15:04:13.993156 26190 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:37057 every 8 connection(s)
I20260926 15:04:13.993539 26124 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:13.994778 26191 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:13.997203 26191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66: Bootstrap starting.
I20260926 15:04:13.997817 26191 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:13.998134 26191 log.cc:824] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:13.999033 26191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66: No bootstrap required, opened a new log
I20260926 15:04:13.999832 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 26124
I20260926 15:04:13.999960 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:14.000182 8600 external_mini_cluster.cc:1428] Setting key 3c1b1981cd7f0cc14195db3d8c625b39
I20260926 15:04:14.001310 26191 raft_consensus.cc:348] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b730f32f73a34733aaafebe90203df66" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37057 } }
I20260926 15:04:14.001518 26191 raft_consensus.cc:374] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:14.001566 26191 raft_consensus.cc:729] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b730f32f73a34733aaafebe90203df66, State: Initialized, Role: FOLLOWER
I20260926 15:04:14.001740 26191 consensus_queue.cc:260] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b730f32f73a34733aaafebe90203df66" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37057 } }
I20260926 15:04:14.001835 26191 raft_consensus.cc:388] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:14.001884 26191 raft_consensus.cc:482] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:14.001940 26191 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:14.002555 26191 raft_consensus.cc:504] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b730f32f73a34733aaafebe90203df66" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37057 } }
I20260926 15:04:14.002714 26191 leader_election.cc:302] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b730f32f73a34733aaafebe90203df66; no voters:
I20260926 15:04:14.002929 26191 leader_election.cc:288] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:14.002982 26196 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:14.003149 26196 raft_consensus.cc:686] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66 [term 1 LEADER]: Becoming Leader. State: Replica: b730f32f73a34733aaafebe90203df66, State: Running, Role: LEADER
I20260926 15:04:14.003253 26191 sys_catalog.cc:564] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:14.003294 26196 consensus_queue.cc:237] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b730f32f73a34733aaafebe90203df66" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37057 } }
I20260926 15:04:14.003700 26196 sys_catalog.cc:455] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b730f32f73a34733aaafebe90203df66. Latest consensus state: current_term: 1 leader_uuid: "b730f32f73a34733aaafebe90203df66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b730f32f73a34733aaafebe90203df66" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37057 } } }
I20260926 15:04:14.003786 26196 sys_catalog.cc:458] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:14.004051 26197 sys_catalog.cc:455] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b730f32f73a34733aaafebe90203df66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b730f32f73a34733aaafebe90203df66" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37057 } } }
I20260926 15:04:14.004226 26197 sys_catalog.cc:458] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:14.005326 26203 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:14.005858 26203 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:14.006847 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:14.000754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33074 (local address 127.8.102.62:37057)
0926 15:04:14.000971 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:14.000979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:14.001004 (+ 25us) server_negotiation.cc:407] Connection header received
0926 15:04:14.001261 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:14.001266 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:14.001322 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:14.001432 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:14.001731 (+ 299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.003261 (+ 1530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.004418 (+ 1157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.004912 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.005026 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:14.005869 (+ 843us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:14.005874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:14.005874 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:14.006097 (+ 223us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:14.006144 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:14.006162 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:14.006429 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:14.006467 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 15:04:14.006559 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":101,"threads_started":1}
I20260926 15:04:14.007448 26203 catalog_manager.cc:1347] Generated new cluster ID: b0ef0f05f0b64a59b9721cabfb6cc4e0
I20260926 15:04:14.007501 26203 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:14.018723 26203 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:14.019310 26203 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:14.026660 26203 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66: Generated new TSK 0
I20260926 15:04:14.026885 26203 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:14.029488 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37057
--builtin_ntp_servers=127.8.102.20:36355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:04:14.113826 26215 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:14.114161 26215 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:14.114198 26215 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:14.115535 26215 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:14.115602 26215 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:14.115643 26215 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:04:14.117117 26215 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26215
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:14.117412 26215 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:14.117764 26215 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 15:04:14.123162 26215 server_base.cc:1034] running on GCE node
W20260926 15:04:14.123126 26220 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:14.123274 26223 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:14.123126 26221 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:14.123605 26215 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:14.123908 26215 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:14.125111 26215 hybrid_clock.cc:648] HybridClock initialized: now 1790435054125109 us; error 52 us; skew 500 ppm
I20260926 15:04:14.126675 26215 webserver.cc:460] Webserver started at http://127.8.102.1:46345/ using document root <none> and password file <none>
I20260926 15:04:14.126991 26215 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:14.127044 26215 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:14.127213 26215 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:14.128423 26215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "e93f63f2836f4c89a661c7cfbc1d3720"
format_stamp: "Formatted at 2026-09-26 15:04:14 on dist-test-slave-jnhs"
server_key: "aab19527a277574ce93dd2c3863639bb"
server_key_iv: "41e154697ad71ed31ec4b372e00c2234"
server_key_version: "encryptionkey@0"
I20260926 15:04:14.128854 26215 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "e93f63f2836f4c89a661c7cfbc1d3720"
format_stamp: "Formatted at 2026-09-26 15:04:14 on dist-test-slave-jnhs"
server_key: "aab19527a277574ce93dd2c3863639bb"
server_key_iv: "41e154697ad71ed31ec4b372e00c2234"
server_key_version: "encryptionkey@0"
I20260926 15:04:14.130399 26215 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:04:14.131364 26229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:14.131541 26215 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:04:14.131665 26215 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "e93f63f2836f4c89a661c7cfbc1d3720"
format_stamp: "Formatted at 2026-09-26 15:04:14 on dist-test-slave-jnhs"
server_key: "aab19527a277574ce93dd2c3863639bb"
server_key_iv: "41e154697ad71ed31ec4b372e00c2234"
server_key_version: "encryptionkey@0"
I20260926 15:04:14.131757 26215 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:14.155543 26215 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:14.155974 26215 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:14.156141 26215 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:14.156414 26215 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:04:14.156787 26215 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:04:14.156823 26215 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:14.156880 26215 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:04:14.156909 26215 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:14.163604 26215 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:41191
I20260926 15:04:14.163654 26342 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:41191 every 8 connection(s)
I20260926 15:04:14.164036 26215 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:04:14.164744 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 26215
I20260926 15:04:14.164855 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:04:14.165126 8600 external_mini_cluster.cc:1428] Setting key 809bbf0d885d7d66c317f8e9ac1c1391
I20260926 15:04:14.166189 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37057
--builtin_ntp_servers=127.8.102.20:36355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 15:04:14.171244 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:14.164905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:33571 (local address 127.8.102.62:37057)
0926 15:04:14.164965 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:14.164969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:14.165272 (+ 303us) server_negotiation.cc:407] Connection header received
0926 15:04:14.166037 (+ 765us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:14.166037 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:14.166048 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:14.166119 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:14.167176 (+ 1057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.167879 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.169567 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.169764 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.169824 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:14.170898 (+ 1074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:14.170900 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:14.170900 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:14.170908 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:14.170925 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:14.170934 (+ 9us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:14.171013 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:14.171130 (+ 117us) server_negotiation.cc:299] Negotiation successful
0926 15:04:14.171162 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 15:04:14.171494 26345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:14.164887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:37057 (local address 127.8.102.1:33571)
0926 15:04:14.165116 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:14.165168 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:14.165922 (+ 754us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:14.166203 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:14.166208 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:14.166221 (+ 13us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:14.167052 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:14.167070 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.167980 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.167981 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:14.169462 (+ 1481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:14.169465 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.169862 (+ 397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.169863 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:14.170048 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:14.170744 (+ 696us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:04:14.170745 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:14.170774 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:04:14.170799 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:14.171012 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:14.171019 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:14.171023 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:04:14.171128 (+ 105us) client_negotiation.cc:240] Negotiation successful
0926 15:04:14.171240 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":84,"threads_started":1}
I20260926 15:04:14.172111 26343 heartbeater.cc:347] Connected to a master server at 127.8.102.62:37057
I20260926 15:04:14.172206 26343 heartbeater.cc:464] Registering TS with master...
I20260926 15:04:14.172408 26343 heartbeater.cc:511] Master 127.8.102.62:37057 requested a full tablet report, sending...
I20260926 15:04:14.172987 26155 ts_manager.cc:194] Registered new tserver with Master: e93f63f2836f4c89a661c7cfbc1d3720 (127.8.102.1:41191)
I20260926 15:04:14.173676 26155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.102.1:33571
W20260926 15:04:14.258879 26346 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:14.259140 26346 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:14.259181 26346 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:14.260406 26346 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:14.260484 26346 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:14.260529 26346 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:04:14.262017 26346 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26346
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:14.262311 26346 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:14.262697 26346 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:14.268784 26351 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:14.268891 26346 server_base.cc:1034] running on GCE node
W20260926 15:04:14.268790 26352 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:14.268785 26354 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:14.269381 26346 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:14.269685 26346 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:14.270874 26346 hybrid_clock.cc:648] HybridClock initialized: now 1790435054270851 us; error 28 us; skew 500 ppm
I20260926 15:04:14.272428 26346 webserver.cc:460] Webserver started at http://127.8.102.2:46039/ using document root <none> and password file <none>
I20260926 15:04:14.272728 26346 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:14.272811 26346 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:14.272979 26346 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:14.274155 26346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "16014c47d8f741e196cf4226097b97fb"
format_stamp: "Formatted at 2026-09-26 15:04:14 on dist-test-slave-jnhs"
server_key: "ccd9a102370bde3fab23630193a06f66"
server_key_iv: "dfb690d1467edd64a1d0cc51b18a1671"
server_key_version: "encryptionkey@0"
I20260926 15:04:14.274552 26346 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "16014c47d8f741e196cf4226097b97fb"
format_stamp: "Formatted at 2026-09-26 15:04:14 on dist-test-slave-jnhs"
server_key: "ccd9a102370bde3fab23630193a06f66"
server_key_iv: "dfb690d1467edd64a1d0cc51b18a1671"
server_key_version: "encryptionkey@0"
I20260926 15:04:14.276053 26346 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:14.276882 26360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:14.277067 26346 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:14.277184 26346 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "16014c47d8f741e196cf4226097b97fb"
format_stamp: "Formatted at 2026-09-26 15:04:14 on dist-test-slave-jnhs"
server_key: "ccd9a102370bde3fab23630193a06f66"
server_key_iv: "dfb690d1467edd64a1d0cc51b18a1671"
server_key_version: "encryptionkey@0"
I20260926 15:04:14.277268 26346 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:14.310036 26346 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:14.310397 26346 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:14.310567 26346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:14.310842 26346 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:04:14.311210 26346 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:04:14.311270 26346 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:14.311314 26346 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:04:14.311347 26346 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:14.317147 26346 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:45521
I20260926 15:04:14.317184 26473 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:45521 every 8 connection(s)
I20260926 15:04:14.317536 26346 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:04:14.322992 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 26346
I20260926 15:04:14.323148 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:04:14.323369 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:14.318314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:49685 (local address 127.8.102.62:37057)
0926 15:04:14.318395 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:14.318397 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:14.318613 (+ 216us) server_negotiation.cc:407] Connection header received
0926 15:04:14.319317 (+ 704us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:14.319318 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:14.319329 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:14.319386 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:14.320091 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.320581 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.321721 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.321923 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.321981 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:14.323011 (+ 1030us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:14.323012 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:14.323013 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:14.323018 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:14.323033 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:14.323037 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:14.323102 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:14.323281 (+ 179us) server_negotiation.cc:299] Negotiation successful
0926 15:04:14.323306 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:04:14.323402 8600 external_mini_cluster.cc:1428] Setting key e6f38b281d21f4158109492bb98a454c
I20260926 15:04:14.323699 26476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:14.318274 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:37057 (local address 127.8.102.2:49685)
0926 15:04:14.318490 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:14.318529 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:14.319182 (+ 653us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:14.319466 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:14.319469 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:14.319479 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:14.319994 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:14.319999 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.320706 (+ 707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.320707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:14.321614 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:14.321618 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.322067 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.322068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:14.322187 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:14.322826 (+ 639us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:04:14.322826 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:14.322870 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:04:14.322902 (+ 32us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:14.323173 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:14.323180 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:14.323183 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:04:14.323269 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 15:04:14.323394 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":151,"thread_start_us":100,"threads_started":1}
I20260926 15:04:14.324348 26474 heartbeater.cc:347] Connected to a master server at 127.8.102.62:37057
I20260926 15:04:14.324450 26474 heartbeater.cc:464] Registering TS with master...
I20260926 15:04:14.324513 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37057
--builtin_ntp_servers=127.8.102.20:36355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 15:04:14.324676 26474 heartbeater.cc:511] Master 127.8.102.62:37057 requested a full tablet report, sending...
I20260926 15:04:14.325060 26155 ts_manager.cc:194] Registered new tserver with Master: 16014c47d8f741e196cf4226097b97fb (127.8.102.2:45521)
I20260926 15:04:14.325469 26155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.102.2:49685
W20260926 15:04:14.408397 26477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:14.408633 26477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:14.408658 26477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:14.409998 26477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:14.410049 26477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:14.410077 26477 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:04:14.411775 26477 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:36355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37057
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:14.412168 26477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:14.412612 26477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:14.418488 26482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:14.418491 26483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:14.418489 26485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:14.418709 26477 server_base.cc:1034] running on GCE node
I20260926 15:04:14.418977 26477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:14.419294 26477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:14.420476 26477 hybrid_clock.cc:648] HybridClock initialized: now 1790435054420431 us; error 60 us; skew 500 ppm
I20260926 15:04:14.421890 26477 webserver.cc:460] Webserver started at http://127.8.102.3:40417/ using document root <none> and password file <none>
I20260926 15:04:14.422132 26477 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:14.422183 26477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:14.422286 26477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:14.423534 26477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "6846e307cd7546b88db543df996625fa"
format_stamp: "Formatted at 2026-09-26 15:04:14 on dist-test-slave-jnhs"
server_key: "973075d3c852d67caab981d1fa0d17bf"
server_key_iv: "4d8ea1a7d9d904cf87da0b1eec846229"
server_key_version: "encryptionkey@0"
I20260926 15:04:14.423870 26477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "6846e307cd7546b88db543df996625fa"
format_stamp: "Formatted at 2026-09-26 15:04:14 on dist-test-slave-jnhs"
server_key: "973075d3c852d67caab981d1fa0d17bf"
server_key_iv: "4d8ea1a7d9d904cf87da0b1eec846229"
server_key_version: "encryptionkey@0"
I20260926 15:04:14.425263 26477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:04:14.426076 26491 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:14.426291 26477 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:14.426402 26477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "6846e307cd7546b88db543df996625fa"
format_stamp: "Formatted at 2026-09-26 15:04:14 on dist-test-slave-jnhs"
server_key: "973075d3c852d67caab981d1fa0d17bf"
server_key_iv: "4d8ea1a7d9d904cf87da0b1eec846229"
server_key_version: "encryptionkey@0"
I20260926 15:04:14.426487 26477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:14.434314 26477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:14.434669 26477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:14.434830 26477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:14.435063 26477 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:04:14.435446 26477 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:04:14.435501 26477 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:14.435542 26477 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:04:14.435570 26477 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:14.441587 26477 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:44017
I20260926 15:04:14.441638 26604 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:44017 every 8 connection(s)
I20260926 15:04:14.441969 26477 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:04:14.447474 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:14.442774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:50247 (local address 127.8.102.62:37057)
0926 15:04:14.442851 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:14.442853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:14.443098 (+ 245us) server_negotiation.cc:407] Connection header received
0926 15:04:14.443665 (+ 567us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:14.443666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:14.443678 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:14.443736 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:14.444441 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.444959 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.446008 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.446201 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.446257 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:14.447243 (+ 986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:14.447244 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:14.447244 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:14.447248 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:14.447261 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:14.447266 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:14.447310 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:14.447386 (+ 76us) server_negotiation.cc:299] Negotiation successful
0926 15:04:14.447412 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 15:04:14.447749 26607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:14.442749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:37057 (local address 127.8.102.3:50247)
0926 15:04:14.442969 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:14.443006 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:14.443532 (+ 526us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:14.443808 (+ 276us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:14.443812 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:14.443820 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:14.444359 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:14.444364 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.445046 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.445047 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:14.445925 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:14.445928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.446280 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.446281 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:14.446402 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:14.447102 (+ 700us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:04:14.447102 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:14.447134 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:04:14.447160 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:14.447313 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:14.447316 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:14.447318 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 15:04:14.447371 (+ 53us) client_negotiation.cc:240] Negotiation successful
0926 15:04:14.447459 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"thread_start_us":110,"threads_started":1}
I20260926 15:04:14.448347 26605 heartbeater.cc:347] Connected to a master server at 127.8.102.62:37057
I20260926 15:04:14.448475 26605 heartbeater.cc:464] Registering TS with master...
I20260926 15:04:14.448680 26605 heartbeater.cc:511] Master 127.8.102.62:37057 requested a full tablet report, sending...
I20260926 15:04:14.449131 26155 ts_manager.cc:194] Registered new tserver with Master: 6846e307cd7546b88db543df996625fa (127.8.102.3:44017)
I20260926 15:04:14.449502 26155 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.102.3:50247
I20260926 15:04:14.449733 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 26477
I20260926 15:04:14.449836 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:04:14.450040 8600 external_mini_cluster.cc:1428] Setting key bd1a5ff9e278fc568093abfbd0273d95
I20260926 15:04:14.451050 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:04:14.456051 26195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:14.452575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33078 (local address 127.8.102.62:37057)
0926 15:04:14.452687 (+ 112us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:14.452690 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:14.452807 (+ 117us) server_negotiation.cc:407] Connection header received
0926 15:04:14.452885 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:14.452886 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:14.452900 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:14.452956 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:14.453140 (+ 184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.453639 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.454473 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.454819 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.454901 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:14.455698 (+ 797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:14.455700 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:14.455700 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:14.455705 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:14.455723 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:14.455727 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:14.455780 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:14.455844 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 15:04:14.455873 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260926 15:04:14.458247 26155 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33078:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:04:14.460032 26155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:04:14.466941 26618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:14.462315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:45521 (local address 127.0.0.1:43642)
0926 15:04:14.462810 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:14.462821 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:14.462894 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:14.463195 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:14.463200 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:14.463203 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:14.463279 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:14.463281 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.464654 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.464655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:14.465811 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:14.465814 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.466601 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.466602 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:14.466708 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:14.466723 (+ 15us) client_negotiation.cc:769] Sending connection context
0926 15:04:14.466802 (+ 79us) client_negotiation.cc:240] Negotiation successful
0926 15:04:14.466859 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"mutex_wait_us":72,"thread_start_us":106,"threads_started":1}
I20260926 15:04:14.466956 26615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:14.462404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.8.102.2:45521)
0926 15:04:14.462557 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:14.462563 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:14.462909 (+ 346us) server_negotiation.cc:407] Connection header received
0926 15:04:14.462994 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:14.462996 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:14.463013 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:14.463090 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:14.463367 (+ 277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.464562 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.465911 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.466508 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.466609 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:14.466737 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:14.466818 (+ 81us) server_negotiation.cc:299] Negotiation successful
0926 15:04:14.466856 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":76,"threads_started":1}
I20260926 15:04:14.467567 26614 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:14.462315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:41191 (local address 127.0.0.1:51658)
0926 15:04:14.462999 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:14.463009 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:14.463061 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:14.463314 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:14.463315 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:14.463317 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:14.463371 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:14.463372 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.464646 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.464647 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:14.466080 (+ 1433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:14.466082 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.467272 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.467273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:14.467345 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:14.467353 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 15:04:14.467411 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 15:04:14.467466 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":448,"thread_start_us":71,"threads_started":1}
I20260926 15:04:14.467772 26617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:14.462583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33318 (local address 127.8.102.3:44017)
0926 15:04:14.462804 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:14.462810 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:14.462824 (+ 14us) server_negotiation.cc:407] Connection header received
0926 15:04:14.462874 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:14.462876 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:14.462895 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:14.463236 (+ 341us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:14.463494 (+ 258us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.464267 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.466195 (+ 1928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.467054 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.467133 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:14.467225 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:14.467612 (+ 387us) server_negotiation.cc:299] Negotiation successful
0926 15:04:14.467663 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":102,"threads_started":1}
I20260926 15:04:14.467568 26613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:14.462315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:44017 (local address 127.0.0.1:33318)
0926 15:04:14.462577 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:14.462650 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:14.462723 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:14.463308 (+ 585us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:14.463309 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:14.463312 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:14.463389 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:14.463392 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.464403 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.464404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:14.466053 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:14.466058 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.467268 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.467269 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:14.467345 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:14.467355 (+ 10us) client_negotiation.cc:769] Sending connection context
0926 15:04:14.467411 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 15:04:14.467479 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":234,"mutex_wait_us":46,"thread_start_us":73,"threads_started":1}
I20260926 15:04:14.468032 26616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:14.462454 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51658 (local address 127.8.102.1:41191)
0926 15:04:14.462721 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:14.462726 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:14.463058 (+ 332us) server_negotiation.cc:407] Connection header received
0926 15:04:14.463154 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:14.463156 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:14.463173 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:14.463246 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:14.463446 (+ 200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.464541 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.466195 (+ 1654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.467054 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.467133 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:14.467239 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:14.467777 (+ 538us) server_negotiation.cc:299] Negotiation successful
0926 15:04:14.467959 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":171,"thread_start_us":135,"threads_started":1}
I20260926 15:04:14.468180 26408 tablet_service.cc:1467] Processing CreateTablet for tablet 4c23afe5050747668dc851a30e87bafa (DEFAULT_TABLE table=test-table [id=beb8d4fe6baa4cecbe54471648aad813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:04:14.468600 26408 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c23afe5050747668dc851a30e87bafa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:14.468602 26539 tablet_service.cc:1467] Processing CreateTablet for tablet 4c23afe5050747668dc851a30e87bafa (DEFAULT_TABLE table=test-table [id=beb8d4fe6baa4cecbe54471648aad813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:04:14.468887 26539 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c23afe5050747668dc851a30e87bafa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:14.468876 26277 tablet_service.cc:1467] Processing CreateTablet for tablet 4c23afe5050747668dc851a30e87bafa (DEFAULT_TABLE table=test-table [id=beb8d4fe6baa4cecbe54471648aad813]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:04:14.469209 26277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c23afe5050747668dc851a30e87bafa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:14.472066 26619 tablet_bootstrap.cc:492] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa: Bootstrap starting.
I20260926 15:04:14.472545 26620 tablet_bootstrap.cc:492] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb: Bootstrap starting.
I20260926 15:04:14.472851 26619 tablet_bootstrap.cc:654] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:14.473145 26621 tablet_bootstrap.cc:492] T 4c23afe5050747668dc851a30e87bafa P e93f63f2836f4c89a661c7cfbc1d3720: Bootstrap starting.
I20260926 15:04:14.473255 26620 tablet_bootstrap.cc:654] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:14.473232 26619 log.cc:824] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:14.473634 26620 log.cc:824] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:14.473907 26621 tablet_bootstrap.cc:654] T 4c23afe5050747668dc851a30e87bafa P e93f63f2836f4c89a661c7cfbc1d3720: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:14.474059 26619 tablet_bootstrap.cc:492] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa: No bootstrap required, opened a new log
I20260926 15:04:14.474151 26619 ts_tablet_manager.cc:1397] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:04:14.474313 26621 log.cc:824] T 4c23afe5050747668dc851a30e87bafa P e93f63f2836f4c89a661c7cfbc1d3720: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:14.474557 26620 tablet_bootstrap.cc:492] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb: No bootstrap required, opened a new log
I20260926 15:04:14.474709 26620 ts_tablet_manager.cc:1397] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:04:14.475342 26621 tablet_bootstrap.cc:492] T 4c23afe5050747668dc851a30e87bafa P e93f63f2836f4c89a661c7cfbc1d3720: No bootstrap required, opened a new log
I20260926 15:04:14.475428 26621 ts_tablet_manager.cc:1397] T 4c23afe5050747668dc851a30e87bafa P e93f63f2836f4c89a661c7cfbc1d3720: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:04:14.476763 26619 raft_consensus.cc:348] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6846e307cd7546b88db543df996625fa" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44017 } } peers { permanent_uuid: "16014c47d8f741e196cf4226097b97fb" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45521 } } peers { permanent_uuid: "e93f63f2836f4c89a661c7cfbc1d3720" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41191 } }
I20260926 15:04:14.476950 26619 raft_consensus.cc:374] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:14.477025 26619 raft_consensus.cc:729] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6846e307cd7546b88db543df996625fa, State: Initialized, Role: FOLLOWER
I20260926 15:04:14.477190 26619 consensus_queue.cc:260] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6846e307cd7546b88db543df996625fa" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44017 } } peers { permanent_uuid: "16014c47d8f741e196cf4226097b97fb" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45521 } } peers { permanent_uuid: "e93f63f2836f4c89a661c7cfbc1d3720" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41191 } }
I20260926 15:04:14.477516 26619 ts_tablet_manager.cc:1428] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:04:14.477680 26605 heartbeater.cc:503] Master 127.8.102.62:37057 was elected leader, sending a full tablet report...
I20260926 15:04:14.477598 26620 raft_consensus.cc:348] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6846e307cd7546b88db543df996625fa" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44017 } } peers { permanent_uuid: "16014c47d8f741e196cf4226097b97fb" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45521 } } peers { permanent_uuid: "e93f63f2836f4c89a661c7cfbc1d3720" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41191 } }
I20260926 15:04:14.478063 26621 raft_consensus.cc:348] T 4c23afe5050747668dc851a30e87bafa P e93f63f2836f4c89a661c7cfbc1d3720 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6846e307cd7546b88db543df996625fa" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44017 } } peers { permanent_uuid: "16014c47d8f741e196cf4226097b97fb" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45521 } } peers { permanent_uuid: "e93f63f2836f4c89a661c7cfbc1d3720" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41191 } }
I20260926 15:04:14.478169 26620 raft_consensus.cc:374] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:14.478216 26621 raft_consensus.cc:374] T 4c23afe5050747668dc851a30e87bafa P e93f63f2836f4c89a661c7cfbc1d3720 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:14.478253 26621 raft_consensus.cc:729] T 4c23afe5050747668dc851a30e87bafa P e93f63f2836f4c89a661c7cfbc1d3720 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e93f63f2836f4c89a661c7cfbc1d3720, State: Initialized, Role: FOLLOWER
I20260926 15:04:14.478277 26620 raft_consensus.cc:729] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16014c47d8f741e196cf4226097b97fb, State: Initialized, Role: FOLLOWER
I20260926 15:04:14.478395 26621 consensus_queue.cc:260] T 4c23afe5050747668dc851a30e87bafa P e93f63f2836f4c89a661c7cfbc1d3720 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6846e307cd7546b88db543df996625fa" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44017 } } peers { permanent_uuid: "16014c47d8f741e196cf4226097b97fb" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45521 } } peers { permanent_uuid: "e93f63f2836f4c89a661c7cfbc1d3720" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41191 } }
I20260926 15:04:14.478557 26620 consensus_queue.cc:260] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6846e307cd7546b88db543df996625fa" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44017 } } peers { permanent_uuid: "16014c47d8f741e196cf4226097b97fb" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45521 } } peers { permanent_uuid: "e93f63f2836f4c89a661c7cfbc1d3720" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41191 } }
I20260926 15:04:14.478690 26621 ts_tablet_manager.cc:1428] T 4c23afe5050747668dc851a30e87bafa P e93f63f2836f4c89a661c7cfbc1d3720: Time spent starting tablet: real 0.003s user 0.000s sys 0.004s
I20260926 15:04:14.478849 26343 heartbeater.cc:503] Master 127.8.102.62:37057 was elected leader, sending a full tablet report...
I20260926 15:04:14.479229 26620 ts_tablet_manager.cc:1428] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 15:04:14.479362 26474 heartbeater.cc:503] Master 127.8.102.62:37057 was elected leader, sending a full tablet report...
W20260926 15:04:14.568115 26475 tablet.cc:2369] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:04:14.577133 26627 raft_consensus.cc:482] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:04:14.577275 26627 raft_consensus.cc:504] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6846e307cd7546b88db543df996625fa" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44017 } } peers { permanent_uuid: "16014c47d8f741e196cf4226097b97fb" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45521 } } peers { permanent_uuid: "e93f63f2836f4c89a661c7cfbc1d3720" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41191 } }
I20260926 15:04:14.577688 26627 leader_election.cc:288] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6846e307cd7546b88db543df996625fa (127.8.102.3:44017), e93f63f2836f4c89a661c7cfbc1d3720 (127.8.102.1:41191)
I20260926 15:04:14.582095 26616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:14.578158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:56117 (local address 127.8.102.1:41191)
0926 15:04:14.578213 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:14.578215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:14.578224 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:04:14.578254 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:14.578255 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:14.578268 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:14.578324 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:14.578653 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.579414 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.581077 (+ 1663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.581690 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.581772 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:14.581782 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:14.581979 (+ 197us) server_negotiation.cc:299] Negotiation successful
0926 15:04:14.582021 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 15:04:14.582096 26617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:14.578042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:49073 (local address 127.8.102.3:44017)
0926 15:04:14.578089 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:14.578091 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:14.578155 (+ 64us) server_negotiation.cc:407] Connection header received
0926 15:04:14.578419 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:14.578421 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:14.578434 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:14.578573 (+ 139us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:14.578808 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.579562 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.581107 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.581688 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.581772 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:14.581782 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:14.581980 (+ 198us) server_negotiation.cc:299] Negotiation successful
0926 15:04:14.582020 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 15:04:14.582113 26476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:14.577895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:41191 (local address 127.8.102.2:56117)
0926 15:04:14.578058 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:14.578074 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:14.578142 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:14.578377 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:14.578378 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:14.578381 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:14.578517 (+ 136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:14.578520 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.579570 (+ 1050us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.579572 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:14.580955 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:14.580958 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.581789 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.581790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:14.581883 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:14.581890 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 15:04:14.581956 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 15:04:14.582011 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":76}
I20260926 15:04:14.582123 26628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:14.577895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:44017 (local address 127.8.102.2:49073)
0926 15:04:14.578090 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:14.578100 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:14.578161 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:14.578592 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:14.578593 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:14.578595 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:14.578726 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:14.578728 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.579635 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.579635 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:14.580966 (+ 1331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:14.580968 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.581789 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.581790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:14.581883 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:14.581890 (+ 7us) client_negotiation.cc:769] Sending connection context
0926 15:04:14.581956 (+ 66us) client_negotiation.cc:240] Negotiation successful
0926 15:04:14.582010 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":179,"mutex_wait_us":28,"thread_start_us":107,"threads_started":1}
I20260926 15:04:14.582521 26297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23afe5050747668dc851a30e87bafa" candidate_uuid: "16014c47d8f741e196cf4226097b97fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e93f63f2836f4c89a661c7cfbc1d3720" is_pre_election: true
I20260926 15:04:14.582546 26559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23afe5050747668dc851a30e87bafa" candidate_uuid: "16014c47d8f741e196cf4226097b97fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6846e307cd7546b88db543df996625fa" is_pre_election: true
I20260926 15:04:14.582751 26297 raft_consensus.cc:2454] T 4c23afe5050747668dc851a30e87bafa P e93f63f2836f4c89a661c7cfbc1d3720 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16014c47d8f741e196cf4226097b97fb in term 0.
I20260926 15:04:14.582762 26559 raft_consensus.cc:2454] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 16014c47d8f741e196cf4226097b97fb in term 0.
I20260926 15:04:14.583029 26364 leader_election.cc:302] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16014c47d8f741e196cf4226097b97fb, 6846e307cd7546b88db543df996625fa; no voters:
I20260926 15:04:14.583216 26627 raft_consensus.cc:2781] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:04:14.583282 26627 raft_consensus.cc:482] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:04:14.583299 26627 raft_consensus.cc:3037] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:14.584064 26627 raft_consensus.cc:504] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6846e307cd7546b88db543df996625fa" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44017 } } peers { permanent_uuid: "16014c47d8f741e196cf4226097b97fb" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45521 } } peers { permanent_uuid: "e93f63f2836f4c89a661c7cfbc1d3720" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41191 } }
I20260926 15:04:14.584223 26627 leader_election.cc:288] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [CANDIDATE]: Term 1 election: Requested vote from peers 6846e307cd7546b88db543df996625fa (127.8.102.3:44017), e93f63f2836f4c89a661c7cfbc1d3720 (127.8.102.1:41191)
I20260926 15:04:14.584372 26559 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23afe5050747668dc851a30e87bafa" candidate_uuid: "16014c47d8f741e196cf4226097b97fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6846e307cd7546b88db543df996625fa"
I20260926 15:04:14.584472 26559 raft_consensus.cc:3037] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:14.585204 26559 raft_consensus.cc:2454] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16014c47d8f741e196cf4226097b97fb in term 1.
I20260926 15:04:14.585346 26297 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c23afe5050747668dc851a30e87bafa" candidate_uuid: "16014c47d8f741e196cf4226097b97fb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e93f63f2836f4c89a661c7cfbc1d3720"
I20260926 15:04:14.585421 26364 leader_election.cc:302] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 16014c47d8f741e196cf4226097b97fb, 6846e307cd7546b88db543df996625fa; no voters:
I20260926 15:04:14.585435 26297 raft_consensus.cc:3037] T 4c23afe5050747668dc851a30e87bafa P e93f63f2836f4c89a661c7cfbc1d3720 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:14.585506 26627 raft_consensus.cc:2781] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:14.585671 26627 raft_consensus.cc:686] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [term 1 LEADER]: Becoming Leader. State: Replica: 16014c47d8f741e196cf4226097b97fb, State: Running, Role: LEADER
I20260926 15:04:14.585899 26627 consensus_queue.cc:237] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6846e307cd7546b88db543df996625fa" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44017 } } peers { permanent_uuid: "16014c47d8f741e196cf4226097b97fb" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45521 } } peers { permanent_uuid: "e93f63f2836f4c89a661c7cfbc1d3720" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41191 } }
I20260926 15:04:14.586426 26297 raft_consensus.cc:2454] T 4c23afe5050747668dc851a30e87bafa P e93f63f2836f4c89a661c7cfbc1d3720 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 16014c47d8f741e196cf4226097b97fb in term 1.
I20260926 15:04:14.586827 26154 catalog_manager.cc:5495] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb reported cstate change: term changed from 0 to 1, leader changed from <none> to 16014c47d8f741e196cf4226097b97fb (127.8.102.2). New cstate: current_term: 1 leader_uuid: "16014c47d8f741e196cf4226097b97fb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6846e307cd7546b88db543df996625fa" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44017 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "16014c47d8f741e196cf4226097b97fb" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45521 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e93f63f2836f4c89a661c7cfbc1d3720" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41191 } health_report { overall_health: UNKNOWN } } }
I20260926 15:04:14.612432 26615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:14.610015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43650 (local address 127.8.102.2:45521)
0926 15:04:14.610080 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:14.610082 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:14.610091 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:04:14.610165 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:14.610166 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:14.610184 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:14.610245 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:04:14.610375 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.610873 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.611652 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:14.611832 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:14.611883 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:14.612129 (+ 246us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:04:14.612228 (+ 99us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:04:14.612293 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:14.612332 (+ 39us) server_negotiation.cc:299] Negotiation successful
0926 15:04:14.612350 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 15:04:14.614254 26559 raft_consensus.cc:1261] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa [term 1 FOLLOWER]: Refusing update from remote peer 16014c47d8f741e196cf4226097b97fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:04:14.614256 26297 raft_consensus.cc:1261] T 4c23afe5050747668dc851a30e87bafa P e93f63f2836f4c89a661c7cfbc1d3720 [term 1 FOLLOWER]: Refusing update from remote peer 16014c47d8f741e196cf4226097b97fb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:04:14.614647 26629 consensus_queue.cc:1035] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "e93f63f2836f4c89a661c7cfbc1d3720" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41191 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:04:14.614773 26627 consensus_queue.cc:1035] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [LEADER]: Connected to new peer: Peer: permanent_uuid: "6846e307cd7546b88db543df996625fa" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44017 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:04:14.617482 26633 mvcc.cc:205] Tried to move back new op lower bound from 7333621983697080320 to 7333621983584497664. Current Snapshot: MvccSnapshot[applied={T|T < 7333621983697080320}]
I20260926 15:04:14.617475 26634 mvcc.cc:205] Tried to move back new op lower bound from 7333621983697080320 to 7333621983584497664. Current Snapshot: MvccSnapshot[applied={T|T < 7333621983697080320}]
I20260926 15:04:14.619781 26155 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:04:14.619882 26155 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33078:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:04:14.620636 26155 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b730f32f73a34733aaafebe90203df66: Sending DeleteTablet for 3 replicas of tablet 4c23afe5050747668dc851a30e87bafa
I20260926 15:04:14.620957 26539 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c23afe5050747668dc851a30e87bafa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:04:14 UTC) from {username='slave'} at 127.0.0.1:33318
I20260926 15:04:14.620998 26408 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c23afe5050747668dc851a30e87bafa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:04:14 UTC) from {username='slave'} at 127.0.0.1:43642
I20260926 15:04:14.621068 26277 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c23afe5050747668dc851a30e87bafa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:04:14 UTC) from {username='slave'} at 127.0.0.1:51658
I20260926 15:04:14.621328 26644 tablet_replica.cc:330] T 4c23afe5050747668dc851a30e87bafa P e93f63f2836f4c89a661c7cfbc1d3720: stopping tablet replica
I20260926 15:04:14.621335 26642 tablet_replica.cc:330] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa: stopping tablet replica
I20260926 15:04:14.621429 26642 raft_consensus.cc:2229] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:04:14.621443 26644 raft_consensus.cc:2229] T 4c23afe5050747668dc851a30e87bafa P e93f63f2836f4c89a661c7cfbc1d3720 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:04:14.621543 26644 raft_consensus.cc:2258] T 4c23afe5050747668dc851a30e87bafa P e93f63f2836f4c89a661c7cfbc1d3720 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:04:14.621543 26642 raft_consensus.cc:2258] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:04:14.621554 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 26215
I20260926 15:04:14.621978 26642 ts_tablet_manager.cc:1905] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:04:14.623368 26643 tablet_replica.cc:330] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb: stopping tablet replica
I20260926 15:04:14.623441 26643 raft_consensus.cc:2229] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:04:14.623523 26643 raft_consensus.cc:2258] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:04:14.623876 26643 ts_tablet_manager.cc:1905] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:04:14.625418 26643 ts_tablet_manager.cc:1918] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:04:14.625469 26643 log.cc:1196] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/4c23afe5050747668dc851a30e87bafa
I20260926 15:04:14.625710 26643 ts_tablet_manager.cc:1939] T 4c23afe5050747668dc851a30e87bafa P 16014c47d8f741e196cf4226097b97fb: Deleting consensus metadata
I20260926 15:04:14.626163 26140 catalog_manager.cc:4841] TS 16014c47d8f741e196cf4226097b97fb (127.8.102.2:45521): tablet 4c23afe5050747668dc851a30e87bafa (table test-table [id=beb8d4fe6baa4cecbe54471648aad813]) successfully deleted
I20260926 15:04:14.626794 26642 ts_tablet_manager.cc:1918] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:04:14.626895 26642 log.cc:1196] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/4c23afe5050747668dc851a30e87bafa
I20260926 15:04:14.627204 26642 ts_tablet_manager.cc:1939] T 4c23afe5050747668dc851a30e87bafa P 6846e307cd7546b88db543df996625fa: Deleting consensus metadata
I20260926 15:04:14.627656 26142 catalog_manager.cc:4841] TS 6846e307cd7546b88db543df996625fa (127.8.102.3:44017): tablet 4c23afe5050747668dc851a30e87bafa (table test-table [id=beb8d4fe6baa4cecbe54471648aad813]) successfully deleted
W20260926 15:04:14.629935 26139 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.102.1:41191 (error 108)
I20260926 15:04:14.630162 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 26346
I20260926 15:04:14.630918 26613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:14.630303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:41191 (local address 127.0.0.1:51668)
0926 15:04:14.630825 (+ 522us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:14.630877 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.1:41191: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":495}
W20260926 15:04:14.631069 26139 catalog_manager.cc:4584] TS e93f63f2836f4c89a661c7cfbc1d3720 (127.8.102.1:41191): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c23afe5050747668dc851a30e87bafa: Network error: Client connection negotiation failed: client connection to 127.8.102.1:41191: connect: Connection refused (error 111)
I20260926 15:04:14.635270 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 26477
I20260926 15:04:14.639612 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 26124
2026-09-26T15:04:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (792 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:14 dist-test-slave-jnhs krb5kdc[26648](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:14 dist-test-slave-jnhs krb5kdc[26648](info): set up 2 sockets
Sep 26 15:04:14 dist-test-slave-jnhs krb5kdc[26648](info): commencing operation
krb5kdc: starting...
W20260926 15:04:15.734723 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.076s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:15 dist-test-slave-jnhs krb5kdc[26648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:04:15.862558 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:37975
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:43651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:37975
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:15.948714 26661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:15.948927 26661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:15.948952 26661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:15.950071 26661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:15.950109 26661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:15.950124 26661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:15.950137 26661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:15.950150 26661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:15.951658 26661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:43651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:37975
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:37975
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26661
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:15.951884 26661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:15.952061 26661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:15.957085 26669 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:15.957149 26666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:15.957152 26667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:15.957221 26661 server_base.cc:1034] running on GCE node
I20260926 15:04:15.957685 26661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:15.958034 26661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:15.959232 26661 hybrid_clock.cc:648] HybridClock initialized: now 1790435055959185 us; error 52 us; skew 500 ppm
Sep 26 15:04:15 dist-test-slave-jnhs krb5kdc[26648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435055, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:15.961961 26661 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:04:15.962435 26661 webserver.cc:460] Webserver started at http://127.8.102.62:46593/ using document root <none> and password file <none>
I20260926 15:04:15.962749 26661 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:15.962807 26661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:15.962968 26661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:15.964097 26661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "ea21ecc992df4b199a4d59b3409839f7"
format_stamp: "Formatted at 2026-09-26 15:04:15 on dist-test-slave-jnhs"
server_key: "f67b0970971ccd1a2385fdbd570f5ad1"
server_key_iv: "07637cba2d525611dbb9ee0af15c0f18"
server_key_version: "encryptionkey@0"
I20260926 15:04:15.964466 26661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "ea21ecc992df4b199a4d59b3409839f7"
format_stamp: "Formatted at 2026-09-26 15:04:15 on dist-test-slave-jnhs"
server_key: "f67b0970971ccd1a2385fdbd570f5ad1"
server_key_iv: "07637cba2d525611dbb9ee0af15c0f18"
server_key_version: "encryptionkey@0"
I20260926 15:04:15.965860 26661 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:04:15.966743 26676 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:15.966936 26661 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:15.967083 26661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "ea21ecc992df4b199a4d59b3409839f7"
format_stamp: "Formatted at 2026-09-26 15:04:15 on dist-test-slave-jnhs"
server_key: "f67b0970971ccd1a2385fdbd570f5ad1"
server_key_iv: "07637cba2d525611dbb9ee0af15c0f18"
server_key_version: "encryptionkey@0"
I20260926 15:04:15.967172 26661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:15.986337 26661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:15.988754 26661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:15.988929 26661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:15.993207 26661 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:37975
I20260926 15:04:15.993355 26728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:37975 every 8 connection(s)
I20260926 15:04:15.993652 26661 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:15.994771 26729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:15.997152 26729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7: Bootstrap starting.
I20260926 15:04:15.997972 26729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:15.997996 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 26661
I20260926 15:04:15.998142 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:15.998394 8600 external_mini_cluster.cc:1428] Setting key dc51235abd36e73009afd7977d2570fb
I20260926 15:04:15.998373 26729 log.cc:824] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:15.999509 26729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7: No bootstrap required, opened a new log
I20260926 15:04:16.001708 26729 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea21ecc992df4b199a4d59b3409839f7" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37975 } }
I20260926 15:04:16.001889 26729 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:16.001941 26729 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea21ecc992df4b199a4d59b3409839f7, State: Initialized, Role: FOLLOWER
I20260926 15:04:16.002079 26729 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea21ecc992df4b199a4d59b3409839f7" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37975 } }
I20260926 15:04:16.002188 26729 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:16.002233 26729 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:16.002285 26729 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:16.002986 26729 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea21ecc992df4b199a4d59b3409839f7" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37975 } }
I20260926 15:04:16.003113 26729 leader_election.cc:302] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea21ecc992df4b199a4d59b3409839f7; no voters:
I20260926 15:04:16.003314 26729 leader_election.cc:288] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:16.003343 26734 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:16.003547 26734 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7 [term 1 LEADER]: Becoming Leader. State: Replica: ea21ecc992df4b199a4d59b3409839f7, State: Running, Role: LEADER
I20260926 15:04:16.003667 26729 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:16.003679 26734 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea21ecc992df4b199a4d59b3409839f7" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37975 } }
I20260926 15:04:16.004163 26736 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea21ecc992df4b199a4d59b3409839f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea21ecc992df4b199a4d59b3409839f7" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37975 } } }
I20260926 15:04:16.004184 26735 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea21ecc992df4b199a4d59b3409839f7. Latest consensus state: current_term: 1 leader_uuid: "ea21ecc992df4b199a4d59b3409839f7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea21ecc992df4b199a4d59b3409839f7" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37975 } } }
I20260926 15:04:16.004382 26735 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:16.004622 26743 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:16.004283 26736 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:16.005182 26743 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 15:04:15 dist-test-slave-jnhs krb5kdc[26648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435055, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:04:16.007423 26743 catalog_manager.cc:1347] Generated new cluster ID: 5c0857bfa61b49e086b01fd4bd47819a
I20260926 15:04:16.007508 26743 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:16.010062 26733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:15.999181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37236 (local address 127.8.102.62:37975)
0926 15:04:15.999385 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:15.999395 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:15.999435 (+ 40us) server_negotiation.cc:407] Connection header received
0926 15:04:15.999998 (+ 563us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:16.000006 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:16.000060 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:16.000217 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:16.000877 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.001772 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.002769 (+ 997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.003295 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.003416 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:16.006354 (+ 2938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:16.006358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:16.006359 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:16.006373 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:16.008284 (+ 1911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:16.008707 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:16.008710 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:16.008711 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:16.008833 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:16.009079 (+ 246us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:16.009080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:16.009081 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:16.009410 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:16.009578 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:16.009660 (+ 82us) server_negotiation.cc:299] Negotiation successful
0926 15:04:16.009810 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":148,"thread_start_us":93,"threads_started":1}
I20260926 15:04:16.015403 26743 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:16.016036 26743 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:16.025750 26743 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7: Generated new TSK 0
I20260926 15:04:16.025995 26743 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
I20260926 15:04:16.058046 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37975
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:43651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:16.141234 26755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:16.141482 26755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:16.141523 26755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:16.142719 26755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:16.142794 26755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:16.142841 26755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:04:16.144322 26755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:43651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26755
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:16.144613 26755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:16.144860 26755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:16.150059 26761 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:16.150097 26763 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:16.150084 26760 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:16.150441 26755 server_base.cc:1034] running on GCE node
I20260926 15:04:16.150704 26755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:16.150995 26755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:16.152176 26755 hybrid_clock.cc:648] HybridClock initialized: now 1790435056152163 us; error 31 us; skew 500 ppm
Sep 26 15:04:16 dist-test-slave-jnhs krb5kdc[26648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:16.155107 26755 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:04:16.155608 26755 webserver.cc:460] Webserver started at http://127.8.102.1:43655/ using document root <none> and password file <none>
I20260926 15:04:16.155872 26755 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:16.155922 26755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:16.156124 26755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:16.157276 26755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "9a78d20b3e7f40a68ed0d804aa29aa82"
format_stamp: "Formatted at 2026-09-26 15:04:16 on dist-test-slave-jnhs"
server_key: "a397c56a6b6fcf72db4014856857c952"
server_key_iv: "db6f70e4f25c32240e46af228c61cab9"
server_key_version: "encryptionkey@0"
I20260926 15:04:16.157663 26755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "9a78d20b3e7f40a68ed0d804aa29aa82"
format_stamp: "Formatted at 2026-09-26 15:04:16 on dist-test-slave-jnhs"
server_key: "a397c56a6b6fcf72db4014856857c952"
server_key_iv: "db6f70e4f25c32240e46af228c61cab9"
server_key_version: "encryptionkey@0"
I20260926 15:04:16.159250 26755 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 15:04:16.160264 26770 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:16.160488 26755 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:04:16.160590 26755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "9a78d20b3e7f40a68ed0d804aa29aa82"
format_stamp: "Formatted at 2026-09-26 15:04:16 on dist-test-slave-jnhs"
server_key: "a397c56a6b6fcf72db4014856857c952"
server_key_iv: "db6f70e4f25c32240e46af228c61cab9"
server_key_version: "encryptionkey@0"
I20260926 15:04:16.160650 26755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:16.171907 26755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:16.174202 26755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:16.174324 26755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:16.174515 26755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:04:16.174898 26755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:04:16.174933 26755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:16.174981 26755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:04:16.175007 26755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:16.180730 26755 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:41725
I20260926 15:04:16.180822 26883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:41725 every 8 connection(s)
I20260926 15:04:16.181150 26755 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:04:16.183902 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 26755
I20260926 15:04:16.184021 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:04:16.184253 8600 external_mini_cluster.cc:1428] Setting key 89bdef404145e558f16a3eaf427de378
Sep 26 15:04:16 dist-test-slave-jnhs krb5kdc[26648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.102.2@KRBTEST.COM; defaulting to no policy
I20260926 15:04:16.193559 26733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:16.182023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:40425 (local address 127.8.102.62:37975)
0926 15:04:16.182174 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:16.182176 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:16.182293 (+ 117us) server_negotiation.cc:407] Connection header received
0926 15:04:16.182908 (+ 615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:16.182910 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:16.182921 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:16.182992 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:16.184283 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.184987 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.186559 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.186972 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.187054 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:16.189776 (+ 2722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:16.189777 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:16.189778 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:16.189786 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:16.191389 (+ 1603us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:16.192016 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:16.192017 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:16.192018 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:16.192089 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:16.192456 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:16.192457 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:16.192457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:16.192615 (+ 158us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:16.192693 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:16.193150 (+ 457us) server_negotiation.cc:299] Negotiation successful
0926 15:04:16.193333 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260926 15:04:16.193569 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:16.182000 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:37975 (local address 127.8.102.1:40425)
0926 15:04:16.182160 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:16.182204 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:16.182794 (+ 590us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:16.183107 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:16.183111 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:16.183754 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:16.184123 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:16.184130 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.185067 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.185068 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:16.186450 (+ 1382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:16.186453 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.187088 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.187089 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:16.187204 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:16.187786 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:16.187795 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:16.189673 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:16.191486 (+ 1813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:16.191495 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:16.191498 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:16.191906 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:16.192168 (+ 262us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:16.192169 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:16.192169 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:16.192372 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:16.192706 (+ 334us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:16.192714 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:16.192912 (+ 198us) client_negotiation.cc:769] Sending connection context
0926 15:04:16.193080 (+ 168us) client_negotiation.cc:240] Negotiation successful
0926 15:04:16.193273 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":87,"threads_started":1}
I20260926 15:04:16.194306 26884 heartbeater.cc:347] Connected to a master server at 127.8.102.62:37975
I20260926 15:04:16.194433 26884 heartbeater.cc:464] Registering TS with master...
I20260926 15:04:16.194691 26884 heartbeater.cc:511] Master 127.8.102.62:37975 requested a full tablet report, sending...
I20260926 15:04:16.195351 26693 ts_manager.cc:194] Registered new tserver with Master: 9a78d20b3e7f40a68ed0d804aa29aa82 (127.8.102.1:41725)
I20260926 15:04:16.196265 26693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:40425
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
I20260926 15:04:16.214968 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37975
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:43651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:16.297089 26889 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:16.297310 26889 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:16.297334 26889 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:16.298564 26889 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:16.298610 26889 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:16.298656 26889 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:04:16.300166 26889 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:43651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26889
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:16.300371 26889 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:16.300562 26889 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 15:04:16.305825 26889 server_base.cc:1034] running on GCE node
W20260926 15:04:16.305848 26894 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:16.305853 26897 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:16.305850 26895 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:16.306365 26889 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:16.306702 26889 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:16.307924 26889 hybrid_clock.cc:648] HybridClock initialized: now 1790435056307914 us; error 42 us; skew 500 ppm
Sep 26 15:04:16 dist-test-slave-jnhs krb5kdc[26648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:16.310600 26889 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:04:16.311147 26889 webserver.cc:460] Webserver started at http://127.8.102.2:43229/ using document root <none> and password file <none>
I20260926 15:04:16.311344 26889 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:16.311385 26889 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:16.311487 26889 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:16.312512 26889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "5f6cfb97b6bd42519dc9ee0a9eb6519a"
format_stamp: "Formatted at 2026-09-26 15:04:16 on dist-test-slave-jnhs"
server_key: "85b7d04583fa97b893093c3cd79d6adf"
server_key_iv: "bcc0752e30a495920c02a5d61486a83a"
server_key_version: "encryptionkey@0"
I20260926 15:04:16.312824 26889 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "5f6cfb97b6bd42519dc9ee0a9eb6519a"
format_stamp: "Formatted at 2026-09-26 15:04:16 on dist-test-slave-jnhs"
server_key: "85b7d04583fa97b893093c3cd79d6adf"
server_key_iv: "bcc0752e30a495920c02a5d61486a83a"
server_key_version: "encryptionkey@0"
I20260926 15:04:16.314236 26889 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 15:04:16.315126 26904 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:16.315351 26889 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 15:04:16.315481 26889 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "5f6cfb97b6bd42519dc9ee0a9eb6519a"
format_stamp: "Formatted at 2026-09-26 15:04:16 on dist-test-slave-jnhs"
server_key: "85b7d04583fa97b893093c3cd79d6adf"
server_key_iv: "bcc0752e30a495920c02a5d61486a83a"
server_key_version: "encryptionkey@0"
I20260926 15:04:16.315583 26889 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:16.326822 26889 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:16.329172 26889 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:16.329370 26889 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:16.329636 26889 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:04:16.329990 26889 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:04:16.330029 26889 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:16.330081 26889 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:04:16.330103 26889 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:16.336129 26889 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:45387
I20260926 15:04:16.336179 27017 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:45387 every 8 connection(s)
I20260926 15:04:16.336514 26889 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:04:16.340207 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 26889
I20260926 15:04:16.340337 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:04:16.340569 8600 external_mini_cluster.cc:1428] Setting key af9dfa6fa9d0bd92b9231616fdb740f5
Sep 26 15:04:16 dist-test-slave-jnhs krb5kdc[26648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:04:16.347421 26733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:16.337273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:39313 (local address 127.8.102.62:37975)
0926 15:04:16.337345 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:16.337347 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:16.337580 (+ 233us) server_negotiation.cc:407] Connection header received
0926 15:04:16.338110 (+ 530us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:16.338112 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:16.338123 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:16.338191 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:16.339260 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.339788 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.341203 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.341409 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.341482 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:16.344158 (+ 2676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:16.344159 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:16.344159 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:16.344167 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:16.345667 (+ 1500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:16.346227 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:16.346228 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:16.346228 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:16.346300 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:16.346680 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:16.346682 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:16.346682 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:16.346819 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:16.346880 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:16.347277 (+ 397us) server_negotiation.cc:299] Negotiation successful
0926 15:04:16.347353 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 15:04:16.347695 27020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:16.337232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:37975 (local address 127.8.102.2:39313)
0926 15:04:16.337415 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:16.337460 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:16.337991 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:16.338295 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:16.338298 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:16.338906 (+ 608us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:16.339153 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:16.339159 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.339895 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.339896 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:16.341137 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:16.341140 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.341527 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.341528 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:16.341690 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:16.342298 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:16.342307 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:16.344064 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:16.345780 (+ 1716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:16.345787 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:16.345789 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:16.346125 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:16.346382 (+ 257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:16.346383 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:16.346384 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:16.346543 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:16.346896 (+ 353us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:16.346903 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:16.347087 (+ 184us) client_negotiation.cc:769] Sending connection context
0926 15:04:16.347221 (+ 134us) client_negotiation.cc:240] Negotiation successful
0926 15:04:16.347423 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":103,"threads_started":1}
I20260926 15:04:16.348264 27018 heartbeater.cc:347] Connected to a master server at 127.8.102.62:37975
I20260926 15:04:16.348376 27018 heartbeater.cc:464] Registering TS with master...
I20260926 15:04:16.348591 27018 heartbeater.cc:511] Master 127.8.102.62:37975 requested a full tablet report, sending...
I20260926 15:04:16.348953 26693 ts_manager.cc:194] Registered new tserver with Master: 5f6cfb97b6bd42519dc9ee0a9eb6519a (127.8.102.2:45387)
I20260926 15:04:16.349329 26693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:39313
WARNING: no policy specified for kudu/127.8.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
I20260926 15:04:16.370198 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37975
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:43651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:16.453305 27023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:16.453517 27023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:16.453542 27023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:16.454649 27023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:16.454690 27023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:16.454716 27023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:04:16.456283 27023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:43651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:37975
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:16.456480 27023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:16.456661 27023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:16.461912 27031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:16.461917 27028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:16.461946 27023 server_base.cc:1034] running on GCE node
W20260926 15:04:16.461930 27029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:16.462472 27023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:16.462776 27023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:16.463930 27023 hybrid_clock.cc:648] HybridClock initialized: now 1790435056463917 us; error 31 us; skew 500 ppm
Sep 26 15:04:16 dist-test-slave-jnhs krb5kdc[26648](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:16.466605 27023 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:04:16.467119 27023 webserver.cc:460] Webserver started at http://127.8.102.3:36949/ using document root <none> and password file <none>
I20260926 15:04:16.467343 27023 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:16.467389 27023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:16.467552 27023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:16.468611 27023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "4bb26f4a6f77431ebb6f7df37851d044"
format_stamp: "Formatted at 2026-09-26 15:04:16 on dist-test-slave-jnhs"
server_key: "f1422b8aad603267cb2dfcf9b32c68d0"
server_key_iv: "471bc9c032f79309bca72cf9147e969f"
server_key_version: "encryptionkey@0"
I20260926 15:04:16.468971 27023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "4bb26f4a6f77431ebb6f7df37851d044"
format_stamp: "Formatted at 2026-09-26 15:04:16 on dist-test-slave-jnhs"
server_key: "f1422b8aad603267cb2dfcf9b32c68d0"
server_key_iv: "471bc9c032f79309bca72cf9147e969f"
server_key_version: "encryptionkey@0"
I20260926 15:04:16.470373 27023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:04:16.471231 27038 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:16.471421 27023 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 15:04:16.471517 27023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "4bb26f4a6f77431ebb6f7df37851d044"
format_stamp: "Formatted at 2026-09-26 15:04:16 on dist-test-slave-jnhs"
server_key: "f1422b8aad603267cb2dfcf9b32c68d0"
server_key_iv: "471bc9c032f79309bca72cf9147e969f"
server_key_version: "encryptionkey@0"
I20260926 15:04:16.471596 27023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:16.481622 27023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:16.484088 27023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:16.484269 27023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:16.484521 27023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:04:16.484865 27023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:04:16.484925 27023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:16.484970 27023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:04:16.485031 27023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:16.491812 27023 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:44641
I20260926 15:04:16.491863 27151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:44641 every 8 connection(s)
I20260926 15:04:16.492256 27023 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:04:16.495694 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 27023
I20260926 15:04:16.495801 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:04:16.496105 8600 external_mini_cluster.cc:1428] Setting key db6801a0874a184de107d6d3990642fa
Sep 26 15:04:16 dist-test-slave-jnhs krb5kdc[26648](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435056, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:04:16.503369 26733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:16.493255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:60621 (local address 127.8.102.62:37975)
0926 15:04:16.493330 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:16.493332 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:16.493462 (+ 130us) server_negotiation.cc:407] Connection header received
0926 15:04:16.493976 (+ 514us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:16.493976 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:16.493986 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:16.494045 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:16.495216 (+ 1171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.495737 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.497511 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.497726 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.497792 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:16.500464 (+ 2672us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:16.500466 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:16.500466 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:16.500474 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:16.501863 (+ 1389us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:16.502303 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:16.502304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:16.502304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:16.502357 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:16.502680 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:16.502681 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:16.502682 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:16.502799 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:16.502855 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:16.503229 (+ 374us) server_negotiation.cc:299] Negotiation successful
0926 15:04:16.503304 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 15:04:16.503473 27154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:16.493140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:37975 (local address 127.8.102.3:60621)
0926 15:04:16.493318 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:16.493367 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:16.493885 (+ 518us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:16.494113 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:16.494117 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:16.494847 (+ 730us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:16.495066 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:16.495072 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.495893 (+ 821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.495895 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:16.497410 (+ 1515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:16.497412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.497823 (+ 411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.497824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:16.497943 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:16.498689 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:16.498701 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:16.500361 (+ 1660us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:16.501952 (+ 1591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:16.501958 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:16.501960 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:16.502225 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:16.502441 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:16.502442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:16.502442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:16.502571 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:16.502881 (+ 310us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:16.502887 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:16.503023 (+ 136us) client_negotiation.cc:769] Sending connection context
0926 15:04:16.503126 (+ 103us) client_negotiation.cc:240] Negotiation successful
0926 15:04:16.503260 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
I20260926 15:04:16.504038 27152 heartbeater.cc:347] Connected to a master server at 127.8.102.62:37975
I20260926 15:04:16.504189 27152 heartbeater.cc:464] Registering TS with master...
I20260926 15:04:16.504402 27152 heartbeater.cc:511] Master 127.8.102.62:37975 requested a full tablet report, sending...
I20260926 15:04:16.504832 26693 ts_manager.cc:194] Registered new tserver with Master: 4bb26f4a6f77431ebb6f7df37851d044 (127.8.102.3:44641)
I20260926 15:04:16.505283 26693 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:60621
I20260926 15:04:16.508152 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:04:16.516091 26733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:16.509523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37252 (local address 127.8.102.62:37975)
0926 15:04:16.509614 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:16.509616 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:16.509628 (+ 12us) server_negotiation.cc:407] Connection header received
0926 15:04:16.509687 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:16.509687 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:16.509696 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:16.509741 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:16.510270 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.510821 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.511523 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.511716 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.511768 (+ 52us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:16.513075 (+ 1307us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:16.513076 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:16.513076 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:16.513083 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:16.514705 (+ 1622us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:16.515079 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:16.515080 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:16.515080 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:16.515145 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:16.515363 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:16.515364 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:16.515364 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:16.515541 (+ 177us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:16.515636 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:16.515816 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 15:04:16.515918 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260926 15:04:16.517799 26693 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:04:16.519580 26693 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:04:16.526242 27161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:16.521954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:41725 (local address 127.0.0.1:45336)
0926 15:04:16.522575 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:16.522582 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:16.522615 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:16.522912 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:16.522913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:16.522916 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:16.522959 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:16.522961 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.524290 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.524291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:16.525249 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:16.525251 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.525937 (+ 686us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.525937 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:16.526016 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:16.526032 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 15:04:16.526103 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 15:04:16.526160 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":612,"mutex_wait_us":5,"thread_start_us":72,"threads_started":1}
I20260926 15:04:16.526252 27163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:16.522193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45336 (local address 127.8.102.1:41725)
0926 15:04:16.522401 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:16.522406 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:16.522664 (+ 258us) server_negotiation.cc:407] Connection header received
0926 15:04:16.522733 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:16.522735 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:16.522750 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:16.522824 (+ 74us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:16.523110 (+ 286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.524212 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.525334 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.525846 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.525937 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:16.526009 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:16.526127 (+ 118us) server_negotiation.cc:299] Negotiation successful
0926 15:04:16.526164 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":163,"thread_start_us":96,"threads_started":1}
I20260926 15:04:16.526716 27162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:16.521952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:45387 (local address 127.0.0.1:49188)
0926 15:04:16.522388 (+ 436us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:16.522399 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:16.522486 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:16.522719 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:16.522722 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:16.522725 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:16.522824 (+ 99us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:16.522827 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.524231 (+ 1404us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.524232 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:16.525315 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:16.525317 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.526409 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.526410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:16.526492 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:16.526496 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:04:16.526561 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 15:04:16.526599 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":422,"mutex_wait_us":32,"thread_start_us":70,"threads_started":1}
I20260926 15:04:16.526762 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:16.522214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49188 (local address 127.8.102.2:45387)
0926 15:04:16.522418 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:16.522422 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:16.522464 (+ 42us) server_negotiation.cc:407] Connection header received
0926 15:04:16.522573 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:16.522574 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:16.522588 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:16.522706 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:16.522984 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.524126 (+ 1142us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.525411 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.526115 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.526311 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:16.526375 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:16.526587 (+ 212us) server_negotiation.cc:299] Negotiation successful
0926 15:04:16.526625 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":176,"thread_start_us":102,"threads_started":1}
I20260926 15:04:16.527220 27160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:16.521952 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:44641 (local address 127.0.0.1:44742)
0926 15:04:16.522082 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:16.522125 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:16.522723 (+ 598us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:16.522989 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:16.522990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:16.522992 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:16.523033 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:16.523034 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.523903 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.523904 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:16.525308 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:16.525311 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.526290 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.526291 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:16.527015 (+ 724us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:16.527019 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:04:16.527108 (+ 89us) client_negotiation.cc:240] Negotiation successful
0926 15:04:16.527157 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":27,"thread_start_us":72,"threads_started":1}
I20260926 15:04:16.527251 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:16.522227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44742 (local address 127.8.102.3:44641)
0926 15:04:16.522419 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:16.522424 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:16.522436 (+ 12us) server_negotiation.cc:407] Connection header received
0926 15:04:16.522814 (+ 378us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:16.522817 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:16.522833 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:16.522904 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:16.523111 (+ 207us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.523792 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.525412 (+ 1620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.526115 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.526196 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:16.526256 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:16.527129 (+ 873us) server_negotiation.cc:299] Negotiation successful
0926 15:04:16.527174 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157,"thread_start_us":81,"threads_started":1}
I20260926 15:04:16.527134 26818 tablet_service.cc:1467] Processing CreateTablet for tablet 09f60608c3084249927bd81bb9c4ecd1 (DEFAULT_TABLE table=test-table [id=5d9ef9cc66e8478ea934053df6c6a912]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:04:16.527542 26818 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09f60608c3084249927bd81bb9c4ecd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:16.528144 27086 tablet_service.cc:1467] Processing CreateTablet for tablet 09f60608c3084249927bd81bb9c4ecd1 (DEFAULT_TABLE table=test-table [id=5d9ef9cc66e8478ea934053df6c6a912]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:04:16.528554 27086 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09f60608c3084249927bd81bb9c4ecd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:16.530292 27166 tablet_bootstrap.cc:492] T 09f60608c3084249927bd81bb9c4ecd1 P 9a78d20b3e7f40a68ed0d804aa29aa82: Bootstrap starting.
I20260926 15:04:16.530952 26952 tablet_service.cc:1467] Processing CreateTablet for tablet 09f60608c3084249927bd81bb9c4ecd1 (DEFAULT_TABLE table=test-table [id=5d9ef9cc66e8478ea934053df6c6a912]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:04:16.531150 27166 tablet_bootstrap.cc:654] T 09f60608c3084249927bd81bb9c4ecd1 P 9a78d20b3e7f40a68ed0d804aa29aa82: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:16.531287 26952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 09f60608c3084249927bd81bb9c4ecd1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:16.531585 27167 tablet_bootstrap.cc:492] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044: Bootstrap starting.
I20260926 15:04:16.531752 27166 log.cc:824] T 09f60608c3084249927bd81bb9c4ecd1 P 9a78d20b3e7f40a68ed0d804aa29aa82: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:16.532254 27167 tablet_bootstrap.cc:654] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:16.532608 27166 tablet_bootstrap.cc:492] T 09f60608c3084249927bd81bb9c4ecd1 P 9a78d20b3e7f40a68ed0d804aa29aa82: No bootstrap required, opened a new log
I20260926 15:04:16.532670 27166 ts_tablet_manager.cc:1397] T 09f60608c3084249927bd81bb9c4ecd1 P 9a78d20b3e7f40a68ed0d804aa29aa82: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:04:16.532672 27167 log.cc:824] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:16.533304 27167 tablet_bootstrap.cc:492] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044: No bootstrap required, opened a new log
I20260926 15:04:16.533346 27167 ts_tablet_manager.cc:1397] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:04:16.534487 27170 tablet_bootstrap.cc:492] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a: Bootstrap starting.
I20260926 15:04:16.534791 27166 raft_consensus.cc:348] T 09f60608c3084249927bd81bb9c4ecd1 P 9a78d20b3e7f40a68ed0d804aa29aa82 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6cfb97b6bd42519dc9ee0a9eb6519a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45387 } } peers { permanent_uuid: "9a78d20b3e7f40a68ed0d804aa29aa82" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41725 } } peers { permanent_uuid: "4bb26f4a6f77431ebb6f7df37851d044" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44641 } }
I20260926 15:04:16.534927 27166 raft_consensus.cc:374] T 09f60608c3084249927bd81bb9c4ecd1 P 9a78d20b3e7f40a68ed0d804aa29aa82 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:16.534948 27166 raft_consensus.cc:729] T 09f60608c3084249927bd81bb9c4ecd1 P 9a78d20b3e7f40a68ed0d804aa29aa82 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a78d20b3e7f40a68ed0d804aa29aa82, State: Initialized, Role: FOLLOWER
I20260926 15:04:16.535068 27166 consensus_queue.cc:260] T 09f60608c3084249927bd81bb9c4ecd1 P 9a78d20b3e7f40a68ed0d804aa29aa82 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6cfb97b6bd42519dc9ee0a9eb6519a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45387 } } peers { permanent_uuid: "9a78d20b3e7f40a68ed0d804aa29aa82" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41725 } } peers { permanent_uuid: "4bb26f4a6f77431ebb6f7df37851d044" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44641 } }
I20260926 15:04:16.535220 27170 tablet_bootstrap.cc:654] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:16.535364 26884 heartbeater.cc:503] Master 127.8.102.62:37975 was elected leader, sending a full tablet report...
I20260926 15:04:16.535316 27167 raft_consensus.cc:348] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6cfb97b6bd42519dc9ee0a9eb6519a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45387 } } peers { permanent_uuid: "9a78d20b3e7f40a68ed0d804aa29aa82" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41725 } } peers { permanent_uuid: "4bb26f4a6f77431ebb6f7df37851d044" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44641 } }
I20260926 15:04:16.535490 27167 raft_consensus.cc:374] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:16.535545 27167 raft_consensus.cc:729] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bb26f4a6f77431ebb6f7df37851d044, State: Initialized, Role: FOLLOWER
I20260926 15:04:16.535697 27167 consensus_queue.cc:260] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6cfb97b6bd42519dc9ee0a9eb6519a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45387 } } peers { permanent_uuid: "9a78d20b3e7f40a68ed0d804aa29aa82" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41725 } } peers { permanent_uuid: "4bb26f4a6f77431ebb6f7df37851d044" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44641 } }
I20260926 15:04:16.535351 27166 ts_tablet_manager.cc:1428] T 09f60608c3084249927bd81bb9c4ecd1 P 9a78d20b3e7f40a68ed0d804aa29aa82: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:04:16.535979 27167 ts_tablet_manager.cc:1428] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:04:16.536070 27152 heartbeater.cc:503] Master 127.8.102.62:37975 was elected leader, sending a full tablet report...
I20260926 15:04:16.536207 27170 log.cc:824] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:16.536958 27170 tablet_bootstrap.cc:492] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a: No bootstrap required, opened a new log
I20260926 15:04:16.537043 27170 ts_tablet_manager.cc:1397] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 15:04:16.538537 27170 raft_consensus.cc:348] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6cfb97b6bd42519dc9ee0a9eb6519a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45387 } } peers { permanent_uuid: "9a78d20b3e7f40a68ed0d804aa29aa82" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41725 } } peers { permanent_uuid: "4bb26f4a6f77431ebb6f7df37851d044" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44641 } }
I20260926 15:04:16.538754 27170 raft_consensus.cc:374] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:16.538805 27170 raft_consensus.cc:729] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f6cfb97b6bd42519dc9ee0a9eb6519a, State: Initialized, Role: FOLLOWER
I20260926 15:04:16.538925 27170 consensus_queue.cc:260] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6cfb97b6bd42519dc9ee0a9eb6519a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45387 } } peers { permanent_uuid: "9a78d20b3e7f40a68ed0d804aa29aa82" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41725 } } peers { permanent_uuid: "4bb26f4a6f77431ebb6f7df37851d044" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44641 } }
I20260926 15:04:16.539204 27170 ts_tablet_manager.cc:1428] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:04:16.539228 27018 heartbeater.cc:503] Master 127.8.102.62:37975 was elected leader, sending a full tablet report...
W20260926 15:04:16.587132 27019 tablet.cc:2369] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:04:16.682037 26885 tablet.cc:2369] T 09f60608c3084249927bd81bb9c4ecd1 P 9a78d20b3e7f40a68ed0d804aa29aa82: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:04:16.742902 27153 tablet.cc:2369] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:04:16.768543 27172 raft_consensus.cc:482] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:04:16.768682 27172 raft_consensus.cc:504] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6cfb97b6bd42519dc9ee0a9eb6519a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45387 } } peers { permanent_uuid: "9a78d20b3e7f40a68ed0d804aa29aa82" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41725 } } peers { permanent_uuid: "4bb26f4a6f77431ebb6f7df37851d044" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44641 } }
I20260926 15:04:16.769061 27172 leader_election.cc:288] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5f6cfb97b6bd42519dc9ee0a9eb6519a (127.8.102.2:45387), 9a78d20b3e7f40a68ed0d804aa29aa82 (127.8.102.1:41725)
I20260926 15:04:16.773128 27163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:16.769378 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:38423 (local address 127.8.102.1:41725)
0926 15:04:16.769435 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:16.769437 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:16.769628 (+ 191us) server_negotiation.cc:407] Connection header received
0926 15:04:16.769743 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:16.769745 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:16.769756 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:16.769815 (+ 59us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:16.769955 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.770733 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.772154 (+ 1421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.772742 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.772827 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:16.772839 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:16.773013 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 15:04:16.773057 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260926 15:04:16.773169 27164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:16.769347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:33289 (local address 127.8.102.2:45387)
0926 15:04:16.769413 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:16.769415 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:16.769428 (+ 13us) server_negotiation.cc:407] Connection header received
0926 15:04:16.769510 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:16.769511 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:16.769529 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:16.769615 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:16.769791 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.770562 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.772155 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.772747 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.772827 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:16.772839 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:16.773059 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 15:04:16.773096 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260926 15:04:16.773154 27154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:16.769264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:45387 (local address 127.8.102.3:33289)
0926 15:04:16.769336 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:16.769348 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:16.769399 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:16.769634 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:16.769636 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:16.769638 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:16.769720 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:16.769723 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.770700 (+ 977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.770701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:16.772031 (+ 1330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:16.772034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.772843 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.772844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:16.772931 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:16.772932 (+ 1us) client_negotiation.cc:769] Sending connection context
0926 15:04:16.772992 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 15:04:16.773043 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40,"mutex_wait_us":46}
I20260926 15:04:16.773161 27175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:16.769268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:41725 (local address 127.8.102.3:38423)
0926 15:04:16.769534 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:16.769542 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:16.769652 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:16.769827 (+ 175us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:16.769827 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:16.769829 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:16.769889 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:16.769890 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.770796 (+ 906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.770796 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:16.772031 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:16.772034 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.772843 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.772844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:16.772926 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:16.772930 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:04:16.772992 (+ 62us) client_negotiation.cc:240] Negotiation successful
0926 15:04:16.773043 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"mutex_wait_us":110,"thread_start_us":108,"threads_started":1}
I20260926 15:04:16.773540 26972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f60608c3084249927bd81bb9c4ecd1" candidate_uuid: "4bb26f4a6f77431ebb6f7df37851d044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f6cfb97b6bd42519dc9ee0a9eb6519a" is_pre_election: true
I20260926 15:04:16.773716 26972 raft_consensus.cc:2454] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bb26f4a6f77431ebb6f7df37851d044 in term 0.
I20260926 15:04:16.773820 26838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f60608c3084249927bd81bb9c4ecd1" candidate_uuid: "4bb26f4a6f77431ebb6f7df37851d044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a78d20b3e7f40a68ed0d804aa29aa82" is_pre_election: true
I20260926 15:04:16.773974 26838 raft_consensus.cc:2454] T 09f60608c3084249927bd81bb9c4ecd1 P 9a78d20b3e7f40a68ed0d804aa29aa82 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bb26f4a6f77431ebb6f7df37851d044 in term 0.
I20260926 15:04:16.773972 27039 leader_election.cc:302] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bb26f4a6f77431ebb6f7df37851d044, 5f6cfb97b6bd42519dc9ee0a9eb6519a; no voters:
I20260926 15:04:16.774147 27172 raft_consensus.cc:2781] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:04:16.774221 27172 raft_consensus.cc:482] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:04:16.774250 27172 raft_consensus.cc:3037] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:16.775143 27172 raft_consensus.cc:504] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6cfb97b6bd42519dc9ee0a9eb6519a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45387 } } peers { permanent_uuid: "9a78d20b3e7f40a68ed0d804aa29aa82" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41725 } } peers { permanent_uuid: "4bb26f4a6f77431ebb6f7df37851d044" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44641 } }
I20260926 15:04:16.775301 27172 leader_election.cc:288] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [CANDIDATE]: Term 1 election: Requested vote from peers 5f6cfb97b6bd42519dc9ee0a9eb6519a (127.8.102.2:45387), 9a78d20b3e7f40a68ed0d804aa29aa82 (127.8.102.1:41725)
I20260926 15:04:16.775460 26838 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f60608c3084249927bd81bb9c4ecd1" candidate_uuid: "4bb26f4a6f77431ebb6f7df37851d044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9a78d20b3e7f40a68ed0d804aa29aa82"
I20260926 15:04:16.775581 26838 raft_consensus.cc:3037] T 09f60608c3084249927bd81bb9c4ecd1 P 9a78d20b3e7f40a68ed0d804aa29aa82 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:16.776369 26838 raft_consensus.cc:2454] T 09f60608c3084249927bd81bb9c4ecd1 P 9a78d20b3e7f40a68ed0d804aa29aa82 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bb26f4a6f77431ebb6f7df37851d044 in term 1.
I20260926 15:04:16.776526 26972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "09f60608c3084249927bd81bb9c4ecd1" candidate_uuid: "4bb26f4a6f77431ebb6f7df37851d044" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f6cfb97b6bd42519dc9ee0a9eb6519a"
I20260926 15:04:16.776580 27041 leader_election.cc:302] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4bb26f4a6f77431ebb6f7df37851d044, 9a78d20b3e7f40a68ed0d804aa29aa82; no voters:
I20260926 15:04:16.776623 26972 raft_consensus.cc:3037] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:16.776680 27172 raft_consensus.cc:2781] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:16.776870 27172 raft_consensus.cc:686] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [term 1 LEADER]: Becoming Leader. State: Replica: 4bb26f4a6f77431ebb6f7df37851d044, State: Running, Role: LEADER
I20260926 15:04:16.777043 27172 consensus_queue.cc:237] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6cfb97b6bd42519dc9ee0a9eb6519a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45387 } } peers { permanent_uuid: "9a78d20b3e7f40a68ed0d804aa29aa82" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41725 } } peers { permanent_uuid: "4bb26f4a6f77431ebb6f7df37851d044" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44641 } }
I20260926 15:04:16.777542 26972 raft_consensus.cc:2454] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bb26f4a6f77431ebb6f7df37851d044 in term 1.
I20260926 15:04:16.778076 26692 catalog_manager.cc:5495] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4bb26f4a6f77431ebb6f7df37851d044 (127.8.102.3). New cstate: current_term: 1 leader_uuid: "4bb26f4a6f77431ebb6f7df37851d044" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f6cfb97b6bd42519dc9ee0a9eb6519a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45387 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9a78d20b3e7f40a68ed0d804aa29aa82" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41725 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4bb26f4a6f77431ebb6f7df37851d044" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 44641 } health_report { overall_health: HEALTHY } } }
I20260926 15:04:16.811331 27165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:16.808802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44750 (local address 127.8.102.3:44641)
0926 15:04:16.808875 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:16.808877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:16.808888 (+ 11us) server_negotiation.cc:407] Connection header received
0926 15:04:16.808935 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:16.808936 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:16.808966 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:16.809048 (+ 82us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:04:16.809183 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.809695 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.810490 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.810729 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.810810 (+ 81us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:16.810955 (+ 145us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:04:16.811063 (+ 108us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:04:16.811180 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:16.811220 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 15:04:16.811229 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 15:04:16.813475 26838 raft_consensus.cc:1261] T 09f60608c3084249927bd81bb9c4ecd1 P 9a78d20b3e7f40a68ed0d804aa29aa82 [term 1 FOLLOWER]: Refusing update from remote peer 4bb26f4a6f77431ebb6f7df37851d044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:04:16.813496 26972 raft_consensus.cc:1261] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a [term 1 FOLLOWER]: Refusing update from remote peer 4bb26f4a6f77431ebb6f7df37851d044: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:04:16.813972 27176 consensus_queue.cc:1035] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9a78d20b3e7f40a68ed0d804aa29aa82" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 41725 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:04:16.814126 27172 consensus_queue.cc:1035] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5f6cfb97b6bd42519dc9ee0a9eb6519a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45387 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:04:16.820022 26691 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:04:16.820130 26691 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37252:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:04:16.820937 26691 catalog_manager.cc:5782] T 00000000000000000000000000000000 P ea21ecc992df4b199a4d59b3409839f7: Sending DeleteTablet for 3 replicas of tablet 09f60608c3084249927bd81bb9c4ecd1
I20260926 15:04:16.821322 26952 tablet_service.cc:1514] Processing DeleteTablet for tablet 09f60608c3084249927bd81bb9c4ecd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:04:16 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:49188
I20260926 15:04:16.821326 26818 tablet_service.cc:1514] Processing DeleteTablet for tablet 09f60608c3084249927bd81bb9c4ecd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:04:16 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:45336
I20260926 15:04:16.821587 27189 tablet_replica.cc:330] T 09f60608c3084249927bd81bb9c4ecd1 P 9a78d20b3e7f40a68ed0d804aa29aa82: stopping tablet replica
I20260926 15:04:16.821573 27086 tablet_service.cc:1514] Processing DeleteTablet for tablet 09f60608c3084249927bd81bb9c4ecd1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:04:16 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:44742
I20260926 15:04:16.821700 27189 raft_consensus.cc:2229] T 09f60608c3084249927bd81bb9c4ecd1 P 9a78d20b3e7f40a68ed0d804aa29aa82 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:04:16.821823 27191 tablet_replica.cc:330] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044: stopping tablet replica
I20260926 15:04:16.821837 27189 raft_consensus.cc:2258] T 09f60608c3084249927bd81bb9c4ecd1 P 9a78d20b3e7f40a68ed0d804aa29aa82 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:04:16.821981 27191 raft_consensus.cc:2229] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:04:16.822153 27191 raft_consensus.cc:2258] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:04:16.822180 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 26755
I20260926 15:04:16.822415 27189 ts_tablet_manager.cc:1905] T 09f60608c3084249927bd81bb9c4ecd1 P 9a78d20b3e7f40a68ed0d804aa29aa82: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:04:16.822873 27190 tablet_replica.cc:330] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a: stopping tablet replica
I20260926 15:04:16.822978 27190 raft_consensus.cc:2229] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:04:16.823086 27190 raft_consensus.cc:2258] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:04:16.824150 27190 ts_tablet_manager.cc:1905] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:04:16.825930 27190 ts_tablet_manager.cc:1918] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:04:16.826035 27190 log.cc:1196] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/09f60608c3084249927bd81bb9c4ecd1
I20260926 15:04:16.826308 27190 ts_tablet_manager.cc:1939] T 09f60608c3084249927bd81bb9c4ecd1 P 5f6cfb97b6bd42519dc9ee0a9eb6519a: Deleting consensus metadata
I20260926 15:04:16.826786 26677 catalog_manager.cc:4841] TS 5f6cfb97b6bd42519dc9ee0a9eb6519a (127.8.102.2:45387): tablet 09f60608c3084249927bd81bb9c4ecd1 (table test-table [id=5d9ef9cc66e8478ea934053df6c6a912]) successfully deleted
W20260926 15:04:16.829511 26679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.102.1:41725 (error 108)
I20260926 15:04:16.829962 27191 ts_tablet_manager.cc:1905] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:04:16.830031 27160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:16.829885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:41725 (local address 127.0.0.1:45348)
0926 15:04:16.829949 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:16.829991 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.1:41725: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 15:04:16.830032 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 26889
W20260926 15:04:16.831576 26679 catalog_manager.cc:4584] TS 9a78d20b3e7f40a68ed0d804aa29aa82 (127.8.102.1:41725): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 09f60608c3084249927bd81bb9c4ecd1: Network error: Client connection negotiation failed: client connection to 127.8.102.1:41725: connect: Connection refused (error 111)
I20260926 15:04:16.831817 27191 ts_tablet_manager.cc:1918] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:04:16.831928 27191 log.cc:1196] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/09f60608c3084249927bd81bb9c4ecd1
I20260926 15:04:16.832242 27191 ts_tablet_manager.cc:1939] T 09f60608c3084249927bd81bb9c4ecd1 P 4bb26f4a6f77431ebb6f7df37851d044: Deleting consensus metadata
I20260926 15:04:16.832782 26678 catalog_manager.cc:4841] TS 4bb26f4a6f77431ebb6f7df37851d044 (127.8.102.3:44641): tablet 09f60608c3084249927bd81bb9c4ecd1 (table test-table [id=5d9ef9cc66e8478ea934053df6c6a912]) successfully deleted
I20260926 15:04:16.836210 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 27023
I20260926 15:04:16.840651 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 26661
2026-09-26T15:04:16Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (2201 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-26T15:04:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:16Z Disabled control of system clock
I20260926 15:04:16.867864 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:42299
--webserver_interface=127.8.102.62
--webserver_port=0
--builtin_ntp_servers=127.8.102.20:41539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:42299
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:04:16.949851 27196 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:16.950084 27196 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:16.950106 27196 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:16.951239 27196 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:16.951279 27196 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:16.951292 27196 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:16.951304 27196 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:16.951316 27196 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:16.952592 27196 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:41539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:42299
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:42299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27196
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:16.952792 27196 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:16.953043 27196 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:16.957948 27202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:16.957957 27201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:16.957958 27204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:16.958431 27196 server_base.cc:1034] running on GCE node
I20260926 15:04:16.958745 27196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:16.959054 27196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:16.960251 27196 hybrid_clock.cc:648] HybridClock initialized: now 1790435056960213 us; error 56 us; skew 500 ppm
I20260926 15:04:16.967599 27196 webserver.cc:460] Webserver started at http://127.8.102.62:39455/ using document root <none> and password file <none>
I20260926 15:04:16.967890 27196 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:16.967965 27196 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:16.968130 27196 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:16.969290 27196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "c5b7154c5fb34b658bef4e4964df247f"
format_stamp: "Formatted at 2026-09-26 15:04:16 on dist-test-slave-jnhs"
server_key: "8ddc0369ff38f7bdd29f3cc91b6177b3"
server_key_iv: "52e011e44931edaf8cf882cdad2fbba0"
server_key_version: "encryptionkey@0"
I20260926 15:04:16.969673 27196 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "c5b7154c5fb34b658bef4e4964df247f"
format_stamp: "Formatted at 2026-09-26 15:04:16 on dist-test-slave-jnhs"
server_key: "8ddc0369ff38f7bdd29f3cc91b6177b3"
server_key_iv: "52e011e44931edaf8cf882cdad2fbba0"
server_key_version: "encryptionkey@0"
I20260926 15:04:16.971230 27196 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 15:04:16.972090 27210 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:16.972290 27196 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:16.972402 27196 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "c5b7154c5fb34b658bef4e4964df247f"
format_stamp: "Formatted at 2026-09-26 15:04:16 on dist-test-slave-jnhs"
server_key: "8ddc0369ff38f7bdd29f3cc91b6177b3"
server_key_iv: "52e011e44931edaf8cf882cdad2fbba0"
server_key_version: "encryptionkey@0"
I20260926 15:04:16.972481 27196 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:16.981980 27196 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:16.982322 27196 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:16.982481 27196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:16.986522 27196 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:42299
I20260926 15:04:16.986562 27262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:42299 every 8 connection(s)
I20260926 15:04:16.986934 27196 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:16.987959 27263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:16.990288 27263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f: Bootstrap starting.
I20260926 15:04:16.990901 27263 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:16.991151 27263 log.cc:824] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:16.991797 27263 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f: No bootstrap required, opened a new log
I20260926 15:04:16.992211 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 27196
I20260926 15:04:16.992340 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:16.992525 8600 external_mini_cluster.cc:1428] Setting key a7f62943d512dd97f8b516e3314b5d99
I20260926 15:04:16.994143 27263 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b7154c5fb34b658bef4e4964df247f" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 42299 } }
I20260926 15:04:16.994323 27263 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:16.994359 27263 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5b7154c5fb34b658bef4e4964df247f, State: Initialized, Role: FOLLOWER
I20260926 15:04:16.994488 27263 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b7154c5fb34b658bef4e4964df247f" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 42299 } }
I20260926 15:04:16.994594 27263 raft_consensus.cc:388] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:16.994628 27263 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:16.994704 27263 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:16.995466 27263 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b7154c5fb34b658bef4e4964df247f" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 42299 } }
I20260926 15:04:16.995574 27263 leader_election.cc:302] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c5b7154c5fb34b658bef4e4964df247f; no voters:
I20260926 15:04:16.995734 27263 leader_election.cc:288] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:16.995812 27268 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:16.995973 27268 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f [term 1 LEADER]: Becoming Leader. State: Replica: c5b7154c5fb34b658bef4e4964df247f, State: Running, Role: LEADER
I20260926 15:04:16.995994 27263 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:16.996061 27268 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b7154c5fb34b658bef4e4964df247f" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 42299 } }
I20260926 15:04:16.996423 27269 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c5b7154c5fb34b658bef4e4964df247f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b7154c5fb34b658bef4e4964df247f" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 42299 } } }
I20260926 15:04:16.996415 27268 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5b7154c5fb34b658bef4e4964df247f. Latest consensus state: current_term: 1 leader_uuid: "c5b7154c5fb34b658bef4e4964df247f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c5b7154c5fb34b658bef4e4964df247f" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 42299 } } }
I20260926 15:04:16.996510 27269 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:16.996516 27268 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f [sys.catalog]: This master's current role is: LEADER
W20260926 15:04:16.997984 27284 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:04:16.998045 27284 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:04:16.998099 27274 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:16.998253 27274 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:16.999871 27274 catalog_manager.cc:1347] Generated new cluster ID: 3cf83c3aab134832889a4cad9ab729d0
I20260926 15:04:16.999926 27274 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:17.000308 27267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:16.993078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55348 (local address 127.8.102.62:42299)
0926 15:04:16.993314 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:16.993325 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:16.993361 (+ 36us) server_negotiation.cc:407] Connection header received
0926 15:04:16.994091 (+ 730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:16.994101 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:16.994188 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:16.994344 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:16.994612 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.996449 (+ 1837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.997695 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:16.998256 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:16.998431 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:16.999436 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:16.999439 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:16.999440 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:16.999666 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:16.999708 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:16.999722 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:17.000053 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:17.000089 (+ 36us) server_negotiation.cc:299] Negotiation successful
0926 15:04:17.000156 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":83,"threads_started":1}
I20260926 15:04:17.010737 27274 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:17.011310 27274 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:17.016112 27274 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f: Generated new TSK 0
I20260926 15:04:17.016343 27274 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:17.024737 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:42299
--builtin_ntp_servers=127.8.102.20:41539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:04:17.123710 27287 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:17.123921 27287 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:17.123944 27287 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:17.125144 27287 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:17.125195 27287 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:17.125222 27287 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:04:17.126626 27287 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:41539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:42299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27287
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:17.126952 27287 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:17.127291 27287 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:17.133301 27292 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:17.133311 27293 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:17.133472 27287 server_base.cc:1034] running on GCE node
W20260926 15:04:17.133582 27295 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:17.133839 27287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:17.134100 27287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:17.135278 27287 hybrid_clock.cc:648] HybridClock initialized: now 1790435057135254 us; error 43 us; skew 500 ppm
I20260926 15:04:17.136592 27287 webserver.cc:460] Webserver started at http://127.8.102.1:43115/ using document root <none> and password file <none>
I20260926 15:04:17.136868 27287 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:17.136945 27287 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:17.137099 27287 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:17.138196 27287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "97cffabceb2c4ccd88bdc03e26bb795c"
format_stamp: "Formatted at 2026-09-26 15:04:17 on dist-test-slave-jnhs"
server_key: "cde70f9ce204aa5cda5773e0b21344d2"
server_key_iv: "de21baf1d52659c17d542718fb9d9161"
server_key_version: "encryptionkey@0"
I20260926 15:04:17.138566 27287 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "97cffabceb2c4ccd88bdc03e26bb795c"
format_stamp: "Formatted at 2026-09-26 15:04:17 on dist-test-slave-jnhs"
server_key: "cde70f9ce204aa5cda5773e0b21344d2"
server_key_iv: "de21baf1d52659c17d542718fb9d9161"
server_key_version: "encryptionkey@0"
I20260926 15:04:17.139961 27287 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:04:17.140797 27301 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:17.140961 27287 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:04:17.141063 27287 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "97cffabceb2c4ccd88bdc03e26bb795c"
format_stamp: "Formatted at 2026-09-26 15:04:17 on dist-test-slave-jnhs"
server_key: "cde70f9ce204aa5cda5773e0b21344d2"
server_key_iv: "de21baf1d52659c17d542718fb9d9161"
server_key_version: "encryptionkey@0"
I20260926 15:04:17.141147 27287 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:17.156286 27287 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:17.156651 27287 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:17.156814 27287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:17.157058 27287 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:04:17.157406 27287 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:04:17.157464 27287 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:17.157505 27287 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:04:17.157536 27287 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:17.163676 27287 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:33183
I20260926 15:04:17.163767 27414 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:33183 every 8 connection(s)
I20260926 15:04:17.164114 27287 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:04:17.169608 27267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:17.164904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:37291 (local address 127.8.102.62:42299)
0926 15:04:17.164971 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:17.164973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:17.165244 (+ 271us) server_negotiation.cc:407] Connection header received
0926 15:04:17.165784 (+ 540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:17.165786 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:17.165799 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:17.165863 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:17.166560 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.167138 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.168198 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.168406 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.168492 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:17.169300 (+ 808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:17.169301 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:17.169302 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:17.169307 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:17.169322 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:17.169326 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:17.169378 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:17.169479 (+ 101us) server_negotiation.cc:299] Negotiation successful
0926 15:04:17.169517 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260926 15:04:17.169832 27417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:17.164912 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:42299 (local address 127.8.102.1:37291)
0926 15:04:17.165106 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:17.165141 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:17.165669 (+ 528us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:17.165940 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:17.165944 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:17.165952 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:17.166453 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:17.166457 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.167233 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.167234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:17.168110 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:17.168113 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.168503 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.168504 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:17.168639 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:17.169157 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:04:17.169157 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:17.169186 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:04:17.169208 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:17.169387 (+ 179us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:17.169391 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:17.169395 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:04:17.169481 (+ 86us) client_negotiation.cc:240] Negotiation successful
0926 15:04:17.169582 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260926 15:04:17.170384 27415 heartbeater.cc:347] Connected to a master server at 127.8.102.62:42299
I20260926 15:04:17.170486 27415 heartbeater.cc:464] Registering TS with master...
I20260926 15:04:17.170675 27415 heartbeater.cc:511] Master 127.8.102.62:42299 requested a full tablet report, sending...
I20260926 15:04:17.171095 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 27287
I20260926 15:04:17.171175 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:04:17.171250 27227 ts_manager.cc:194] Registered new tserver with Master: 97cffabceb2c4ccd88bdc03e26bb795c (127.8.102.1:33183)
I20260926 15:04:17.171413 8600 external_mini_cluster.cc:1428] Setting key e7cd25b6c82e8076f07d59ca98396ef8
I20260926 15:04:17.172222 27227 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.102.1:37291
I20260926 15:04:17.172786 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:42299
--builtin_ntp_servers=127.8.102.20:41539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:04:17.256488 27418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:17.256735 27418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:17.256773 27418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:17.257946 27418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:17.258013 27418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:17.258073 27418 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:04:17.259567 27418 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:41539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:42299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27418
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:17.259837 27418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:17.260145 27418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:17.265430 27424 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:17.265420 27426 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:17.265403 27423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:17.265655 27418 server_base.cc:1034] running on GCE node
I20260926 15:04:17.265921 27418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:17.266170 27418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:17.267342 27418 hybrid_clock.cc:648] HybridClock initialized: now 1790435057267326 us; error 34 us; skew 500 ppm
I20260926 15:04:17.268729 27418 webserver.cc:460] Webserver started at http://127.8.102.2:37519/ using document root <none> and password file <none>
I20260926 15:04:17.269003 27418 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:17.269086 27418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:17.269244 27418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:17.270375 27418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "a2ad1228d2804836b38784b51ca8b9af"
format_stamp: "Formatted at 2026-09-26 15:04:17 on dist-test-slave-jnhs"
server_key: "162180efe06ba046d6722375ea208534"
server_key_iv: "d3e89243d0d3721d243ee371a5b485fc"
server_key_version: "encryptionkey@0"
I20260926 15:04:17.270792 27418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "a2ad1228d2804836b38784b51ca8b9af"
format_stamp: "Formatted at 2026-09-26 15:04:17 on dist-test-slave-jnhs"
server_key: "162180efe06ba046d6722375ea208534"
server_key_iv: "d3e89243d0d3721d243ee371a5b485fc"
server_key_version: "encryptionkey@0"
I20260926 15:04:17.272197 27418 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:04:17.273087 27432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:17.273393 27418 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:04:17.273535 27418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "a2ad1228d2804836b38784b51ca8b9af"
format_stamp: "Formatted at 2026-09-26 15:04:17 on dist-test-slave-jnhs"
server_key: "162180efe06ba046d6722375ea208534"
server_key_iv: "d3e89243d0d3721d243ee371a5b485fc"
server_key_version: "encryptionkey@0"
I20260926 15:04:17.273630 27418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:17.281464 27418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:17.281777 27418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:17.281931 27418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:17.282168 27418 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:04:17.282505 27418 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:04:17.282557 27418 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:17.282593 27418 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:04:17.282621 27418 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:17.288547 27418 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:45591
I20260926 15:04:17.288591 27545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:45591 every 8 connection(s)
I20260926 15:04:17.288924 27418 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:04:17.294394 27267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:17.289691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:57713 (local address 127.8.102.62:42299)
0926 15:04:17.289781 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:17.289783 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:17.289984 (+ 201us) server_negotiation.cc:407] Connection header received
0926 15:04:17.290543 (+ 559us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:17.290544 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:17.290557 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:17.290607 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:17.291363 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.291881 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.292886 (+ 1005us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.293116 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.293198 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:17.294042 (+ 844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:17.294044 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:17.294044 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:17.294049 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:17.294114 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:17.294119 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:17.294176 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:17.294283 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 15:04:17.294312 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 15:04:17.294679 27548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:17.289654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:42299 (local address 127.8.102.2:57713)
0926 15:04:17.289863 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:17.289896 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:17.290427 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:17.290740 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:17.290743 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:17.290751 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:17.291261 (+ 510us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:17.291266 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.291983 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.291983 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:17.292787 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:17.292789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.293206 (+ 417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.293207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:17.293342 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:17.293894 (+ 552us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:04:17.293894 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:17.293923 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:04:17.293946 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:17.294190 (+ 244us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:17.294197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:17.294200 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:04:17.294285 (+ 85us) client_negotiation.cc:240] Negotiation successful
0926 15:04:17.294391 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":89,"threads_started":1}
I20260926 15:04:17.295297 27546 heartbeater.cc:347] Connected to a master server at 127.8.102.62:42299
I20260926 15:04:17.295434 27546 heartbeater.cc:464] Registering TS with master...
I20260926 15:04:17.295639 27546 heartbeater.cc:511] Master 127.8.102.62:42299 requested a full tablet report, sending...
I20260926 15:04:17.296041 27227 ts_manager.cc:194] Registered new tserver with Master: a2ad1228d2804836b38784b51ca8b9af (127.8.102.2:45591)
I20260926 15:04:17.296480 27227 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.102.2:57713
I20260926 15:04:17.297950 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 27418
I20260926 15:04:17.298135 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:04:17.298312 8600 external_mini_cluster.cc:1428] Setting key 3c0baac5ca418a6cfc58095fc00aaf1e
I20260926 15:04:17.299503 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:42299
--builtin_ntp_servers=127.8.102.20:41539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:04:17.382488 27549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:17.382807 27549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:17.382849 27549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:17.384087 27549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:17.384166 27549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:17.384209 27549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:04:17.385628 27549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:41539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:42299
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27549
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:17.385906 27549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:17.386228 27549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 15:04:17.391695 27549 server_base.cc:1034] running on GCE node
W20260926 15:04:17.391659 27555 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:17.391640 27554 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:17.391669 27557 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:17.392086 27549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:17.392346 27549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:17.393530 27549 hybrid_clock.cc:648] HybridClock initialized: now 1790435057393515 us; error 29 us; skew 500 ppm
I20260926 15:04:17.394944 27549 webserver.cc:460] Webserver started at http://127.8.102.3:33439/ using document root <none> and password file <none>
I20260926 15:04:17.395174 27549 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:17.395219 27549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:17.395310 27549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:17.396477 27549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "26ca93c0aa7a454eabac2f0ba43990b9"
format_stamp: "Formatted at 2026-09-26 15:04:17 on dist-test-slave-jnhs"
server_key: "2f6a0fcc3617820faa753d473e87967d"
server_key_iv: "05123dc6366c62b774329ad2ab8a556f"
server_key_version: "encryptionkey@0"
I20260926 15:04:17.396843 27549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "26ca93c0aa7a454eabac2f0ba43990b9"
format_stamp: "Formatted at 2026-09-26 15:04:17 on dist-test-slave-jnhs"
server_key: "2f6a0fcc3617820faa753d473e87967d"
server_key_iv: "05123dc6366c62b774329ad2ab8a556f"
server_key_version: "encryptionkey@0"
I20260926 15:04:17.398254 27549 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:04:17.399590 27563 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:17.399797 27549 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 15:04:17.399866 27549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "26ca93c0aa7a454eabac2f0ba43990b9"
format_stamp: "Formatted at 2026-09-26 15:04:17 on dist-test-slave-jnhs"
server_key: "2f6a0fcc3617820faa753d473e87967d"
server_key_iv: "05123dc6366c62b774329ad2ab8a556f"
server_key_version: "encryptionkey@0"
I20260926 15:04:17.399957 27549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:17.413550 27549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:17.413872 27549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:17.414022 27549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:17.414265 27549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:04:17.414603 27549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:04:17.414695 27549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:17.414742 27549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:04:17.414767 27549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:17.420709 27549 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:34783
I20260926 15:04:17.420758 27676 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:34783 every 8 connection(s)
I20260926 15:04:17.421103 27549 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:04:17.424750 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 27549
I20260926 15:04:17.424840 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:04:17.425045 8600 external_mini_cluster.cc:1428] Setting key 054025e61c3da825805f176d14adbc57
I20260926 15:04:17.426865 27267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:17.421837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:52007 (local address 127.8.102.62:42299)
0926 15:04:17.421928 (+ 91us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:17.421930 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:17.422122 (+ 192us) server_negotiation.cc:407] Connection header received
0926 15:04:17.422726 (+ 604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:17.422728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:17.422742 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:17.422827 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:17.423528 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.424023 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.425153 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.425448 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.425515 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:17.426532 (+ 1017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:17.426533 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:17.426534 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:17.426538 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:17.426553 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:17.426557 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:17.426612 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:17.426779 (+ 167us) server_negotiation.cc:299] Negotiation successful
0926 15:04:17.426804 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260926 15:04:17.427079 27679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:17.421798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:42299 (local address 127.8.102.3:52007)
0926 15:04:17.421985 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:17.422024 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:17.422519 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:17.422911 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:17.422914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:17.422923 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:17.423440 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:17.423444 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.424110 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.424111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:17.425041 (+ 930us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:17.425044 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.425567 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.425568 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:17.425759 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:17.426377 (+ 618us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:04:17.426378 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:17.426407 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:04:17.426432 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:17.426621 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:17.426625 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:17.426627 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 15:04:17.426754 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 15:04:17.426839 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":98,"threads_started":1}
I20260926 15:04:17.427639 27677 heartbeater.cc:347] Connected to a master server at 127.8.102.62:42299
I20260926 15:04:17.427726 27677 heartbeater.cc:464] Registering TS with master...
I20260926 15:04:17.427929 27677 heartbeater.cc:511] Master 127.8.102.62:42299 requested a full tablet report, sending...
I20260926 15:04:17.428328 27227 ts_manager.cc:194] Registered new tserver with Master: 26ca93c0aa7a454eabac2f0ba43990b9 (127.8.102.3:34783)
I20260926 15:04:17.428702 27227 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.102.3:52007
I20260926 15:04:17.436774 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:04:17.441660 27267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:17.438323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55364 (local address 127.8.102.62:42299)
0926 15:04:17.438407 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:17.438420 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:17.438519 (+ 99us) server_negotiation.cc:407] Connection header received
0926 15:04:17.438563 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:17.438564 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:17.438573 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:17.438627 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:17.438837 (+ 210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.439334 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.440118 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.440301 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.440357 (+ 56us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:17.441182 (+ 825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:17.441184 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:17.441184 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:17.441188 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:17.441204 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:17.441207 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:17.441295 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:17.441539 (+ 244us) server_negotiation.cc:299] Negotiation successful
0926 15:04:17.441554 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260926 15:04:17.443542 27227 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55364:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:04:17.445263 27227 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:04:17.452562 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:17.448333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35430 (local address 127.8.102.1:33183)
0926 15:04:17.448546 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:17.448550 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:17.448560 (+ 10us) server_negotiation.cc:407] Connection header received
0926 15:04:17.448594 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:17.448596 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:17.448608 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:17.448714 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:17.448890 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.450067 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.451458 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.452144 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.452208 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:17.452281 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:17.452432 (+ 151us) server_negotiation.cc:299] Negotiation successful
0926 15:04:17.452475 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
I20260926 15:04:17.452584 27686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:17.448030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:45591 (local address 127.0.0.1:48840)
0926 15:04:17.448274 (+ 244us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:17.448317 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:17.448390 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:17.448920 (+ 530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:17.448921 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:17.448923 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:17.448958 (+ 35us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:17.448959 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.450183 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.450184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:17.451209 (+ 1025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:17.451212 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.452215 (+ 1003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.452216 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:17.452306 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:17.452322 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 15:04:17.452404 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 15:04:17.452484 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":2,"thread_start_us":115,"threads_started":1}
I20260926 15:04:17.452601 27685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:17.448059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:33183 (local address 127.0.0.1:35430)
0926 15:04:17.448274 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:17.448317 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:17.448390 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:17.448720 (+ 330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:17.448723 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:17.448726 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:17.448802 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:17.448805 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.450160 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.450161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:17.451369 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:17.451370 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.452232 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.452233 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:17.452306 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:17.452322 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 15:04:17.452404 (+ 82us) client_negotiation.cc:240] Negotiation successful
0926 15:04:17.452491 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":199,"mutex_wait_us":125,"thread_start_us":95,"threads_started":1}
I20260926 15:04:17.453572 27687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:17.448103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:34783 (local address 127.0.0.1:38680)
0926 15:04:17.448498 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:17.448505 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:17.448546 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:17.449058 (+ 512us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:17.449059 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:17.449061 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:17.449098 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:17.449099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.449977 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.449978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:17.451365 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:17.451368 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.453348 (+ 1980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.453349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:17.453415 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:17.453419 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:04:17.453474 (+ 55us) client_negotiation.cc:240] Negotiation successful
0926 15:04:17.453514 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":376,"mutex_wait_us":85,"thread_start_us":62,"threads_started":1}
I20260926 15:04:17.453595 27689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:17.448332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38680 (local address 127.8.102.3:34783)
0926 15:04:17.448843 (+ 511us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:17.448847 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:17.448856 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:04:17.448895 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:17.448897 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:17.448911 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:17.448983 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:17.449182 (+ 199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.449878 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.452324 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.453262 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.453339 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:17.453429 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:17.453483 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 15:04:17.453519 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":71,"threads_started":1}
I20260926 15:04:17.453776 27349 tablet_service.cc:1467] Processing CreateTablet for tablet e0e546146fab4e2f9035c4466e81bf5f (DEFAULT_TABLE table=test-table [id=82fd7ed4ad5b4559b7318d1130a9781a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:04:17.454185 27349 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e546146fab4e2f9035c4466e81bf5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:17.454416 27611 tablet_service.cc:1467] Processing CreateTablet for tablet e0e546146fab4e2f9035c4466e81bf5f (DEFAULT_TABLE table=test-table [id=82fd7ed4ad5b4559b7318d1130a9781a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:04:17.454778 27611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e546146fab4e2f9035c4466e81bf5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:17.455184 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:17.448114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48840 (local address 127.8.102.2:45591)
0926 15:04:17.448705 (+ 591us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:17.448710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:17.448720 (+ 10us) server_negotiation.cc:407] Connection header received
0926 15:04:17.448755 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:17.448757 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:17.448772 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:17.448849 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:17.449034 (+ 185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.450096 (+ 1062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.451310 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.452089 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.452177 (+ 88us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:17.452281 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:17.454997 (+ 2716us) server_negotiation.cc:299] Negotiation successful
0926 15:04:17.455066 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":75,"threads_started":1}
I20260926 15:04:17.457449 27691 tablet_bootstrap.cc:492] T e0e546146fab4e2f9035c4466e81bf5f P 97cffabceb2c4ccd88bdc03e26bb795c: Bootstrap starting.
I20260926 15:04:17.457931 27692 tablet_bootstrap.cc:492] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9: Bootstrap starting.
I20260926 15:04:17.458218 27480 tablet_service.cc:1467] Processing CreateTablet for tablet e0e546146fab4e2f9035c4466e81bf5f (DEFAULT_TABLE table=test-table [id=82fd7ed4ad5b4559b7318d1130a9781a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:04:17.458396 27691 tablet_bootstrap.cc:654] T e0e546146fab4e2f9035c4466e81bf5f P 97cffabceb2c4ccd88bdc03e26bb795c: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:17.458555 27692 tablet_bootstrap.cc:654] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:17.458614 27480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e0e546146fab4e2f9035c4466e81bf5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:17.458756 27691 log.cc:824] T e0e546146fab4e2f9035c4466e81bf5f P 97cffabceb2c4ccd88bdc03e26bb795c: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:17.458945 27692 log.cc:824] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:17.459573 27691 tablet_bootstrap.cc:492] T e0e546146fab4e2f9035c4466e81bf5f P 97cffabceb2c4ccd88bdc03e26bb795c: No bootstrap required, opened a new log
I20260926 15:04:17.459651 27691 ts_tablet_manager.cc:1397] T e0e546146fab4e2f9035c4466e81bf5f P 97cffabceb2c4ccd88bdc03e26bb795c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260926 15:04:17.462332 27695 tablet_bootstrap.cc:492] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af: Bootstrap starting.
I20260926 15:04:17.462419 27691 raft_consensus.cc:348] T e0e546146fab4e2f9035c4466e81bf5f P 97cffabceb2c4ccd88bdc03e26bb795c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ad1228d2804836b38784b51ca8b9af" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45591 } } peers { permanent_uuid: "26ca93c0aa7a454eabac2f0ba43990b9" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34783 } } peers { permanent_uuid: "97cffabceb2c4ccd88bdc03e26bb795c" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33183 } }
I20260926 15:04:17.462597 27691 raft_consensus.cc:374] T e0e546146fab4e2f9035c4466e81bf5f P 97cffabceb2c4ccd88bdc03e26bb795c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:17.462661 27691 raft_consensus.cc:729] T e0e546146fab4e2f9035c4466e81bf5f P 97cffabceb2c4ccd88bdc03e26bb795c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cffabceb2c4ccd88bdc03e26bb795c, State: Initialized, Role: FOLLOWER
I20260926 15:04:17.462785 27691 consensus_queue.cc:260] T e0e546146fab4e2f9035c4466e81bf5f P 97cffabceb2c4ccd88bdc03e26bb795c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ad1228d2804836b38784b51ca8b9af" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45591 } } peers { permanent_uuid: "26ca93c0aa7a454eabac2f0ba43990b9" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34783 } } peers { permanent_uuid: "97cffabceb2c4ccd88bdc03e26bb795c" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33183 } }
I20260926 15:04:17.463081 27691 ts_tablet_manager.cc:1428] T e0e546146fab4e2f9035c4466e81bf5f P 97cffabceb2c4ccd88bdc03e26bb795c: Time spent starting tablet: real 0.003s user 0.001s sys 0.003s
I20260926 15:04:17.463152 27415 heartbeater.cc:503] Master 127.8.102.62:42299 was elected leader, sending a full tablet report...
I20260926 15:04:17.463240 27695 tablet_bootstrap.cc:654] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:17.463618 27695 log.cc:824] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:17.463985 27692 tablet_bootstrap.cc:492] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9: No bootstrap required, opened a new log
I20260926 15:04:17.464129 27692 ts_tablet_manager.cc:1397] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.000s
I20260926 15:04:17.464517 27695 tablet_bootstrap.cc:492] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af: No bootstrap required, opened a new log
I20260926 15:04:17.464600 27695 ts_tablet_manager.cc:1397] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:04:17.466110 27692 raft_consensus.cc:348] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ad1228d2804836b38784b51ca8b9af" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45591 } } peers { permanent_uuid: "26ca93c0aa7a454eabac2f0ba43990b9" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34783 } } peers { permanent_uuid: "97cffabceb2c4ccd88bdc03e26bb795c" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33183 } }
I20260926 15:04:17.466274 27692 raft_consensus.cc:374] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:17.466322 27692 raft_consensus.cc:729] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26ca93c0aa7a454eabac2f0ba43990b9, State: Initialized, Role: FOLLOWER
I20260926 15:04:17.466442 27692 consensus_queue.cc:260] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ad1228d2804836b38784b51ca8b9af" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45591 } } peers { permanent_uuid: "26ca93c0aa7a454eabac2f0ba43990b9" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34783 } } peers { permanent_uuid: "97cffabceb2c4ccd88bdc03e26bb795c" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33183 } }
I20260926 15:04:17.466427 27695 raft_consensus.cc:348] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ad1228d2804836b38784b51ca8b9af" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45591 } } peers { permanent_uuid: "26ca93c0aa7a454eabac2f0ba43990b9" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34783 } } peers { permanent_uuid: "97cffabceb2c4ccd88bdc03e26bb795c" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33183 } }
I20260926 15:04:17.466578 27695 raft_consensus.cc:374] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:17.466622 27695 raft_consensus.cc:729] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2ad1228d2804836b38784b51ca8b9af, State: Initialized, Role: FOLLOWER
I20260926 15:04:17.466750 27692 ts_tablet_manager.cc:1428] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 15:04:17.466797 27677 heartbeater.cc:503] Master 127.8.102.62:42299 was elected leader, sending a full tablet report...
I20260926 15:04:17.466806 27695 consensus_queue.cc:260] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ad1228d2804836b38784b51ca8b9af" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45591 } } peers { permanent_uuid: "26ca93c0aa7a454eabac2f0ba43990b9" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34783 } } peers { permanent_uuid: "97cffabceb2c4ccd88bdc03e26bb795c" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33183 } }
I20260926 15:04:17.467108 27695 ts_tablet_manager.cc:1428] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260926 15:04:17.467156 27546 heartbeater.cc:503] Master 127.8.102.62:42299 was elected leader, sending a full tablet report...
I20260926 15:04:17.471729 27698 raft_consensus.cc:482] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:04:17.471810 27698 raft_consensus.cc:504] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ad1228d2804836b38784b51ca8b9af" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45591 } } peers { permanent_uuid: "26ca93c0aa7a454eabac2f0ba43990b9" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34783 } } peers { permanent_uuid: "97cffabceb2c4ccd88bdc03e26bb795c" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33183 } }
I20260926 15:04:17.472160 27698 leader_election.cc:288] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a2ad1228d2804836b38784b51ca8b9af (127.8.102.2:45591), 97cffabceb2c4ccd88bdc03e26bb795c (127.8.102.1:33183)
I20260926 15:04:17.475804 27688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:17.472405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:43659 (local address 127.8.102.2:45591)
0926 15:04:17.472450 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:17.472452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:17.472461 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:04:17.472488 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:17.472489 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:17.472498 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:17.472562 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:17.472723 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.473473 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.474865 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.475466 (+ 601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.475545 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:17.475553 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:17.475695 (+ 142us) server_negotiation.cc:299] Negotiation successful
0926 15:04:17.475734 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 15:04:17.475804 27690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:17.472475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:33289 (local address 127.8.102.1:33183)
0926 15:04:17.472528 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:17.472530 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:17.472626 (+ 96us) server_negotiation.cc:407] Connection header received
0926 15:04:17.472707 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:17.472708 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:17.472720 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:17.472770 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:17.472953 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.473624 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.474891 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.475436 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.475534 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:17.475546 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:17.475696 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 15:04:17.475734 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260926 15:04:17.475824 27679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:17.472288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:45591 (local address 127.8.102.3:43659)
0926 15:04:17.472334 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:17.472349 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:17.472396 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:17.472580 (+ 184us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:17.472580 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:17.472582 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:17.472647 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:17.472649 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.473552 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.473553 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:17.474765 (+ 1212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:17.474769 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.475540 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.475541 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:17.475617 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:17.475621 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:04:17.475681 (+ 60us) client_negotiation.cc:240] Negotiation successful
0926 15:04:17.475736 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25,"mutex_wait_us":7}
I20260926 15:04:17.475837 27700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:17.472331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:33183 (local address 127.8.102.3:33289)
0926 15:04:17.472571 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:17.472583 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:17.472636 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:17.472771 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:17.472772 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:17.472774 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:17.472837 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:17.472838 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.473693 (+ 855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.473693 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:17.474817 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:17.474819 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.475567 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.475567 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:17.475630 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:17.475633 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:04:17.475681 (+ 48us) client_negotiation.cc:240] Negotiation successful
0926 15:04:17.475734 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":222,"mutex_wait_us":11,"thread_start_us":110,"threads_started":1}
I20260926 15:04:17.476213 27500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e546146fab4e2f9035c4466e81bf5f" candidate_uuid: "26ca93c0aa7a454eabac2f0ba43990b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2ad1228d2804836b38784b51ca8b9af" is_pre_election: true
I20260926 15:04:17.476279 27369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e546146fab4e2f9035c4466e81bf5f" candidate_uuid: "26ca93c0aa7a454eabac2f0ba43990b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cffabceb2c4ccd88bdc03e26bb795c" is_pre_election: true
I20260926 15:04:17.476393 27369 raft_consensus.cc:2454] T e0e546146fab4e2f9035c4466e81bf5f P 97cffabceb2c4ccd88bdc03e26bb795c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26ca93c0aa7a454eabac2f0ba43990b9 in term 0.
I20260926 15:04:17.476393 27500 raft_consensus.cc:2454] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 26ca93c0aa7a454eabac2f0ba43990b9 in term 0.
I20260926 15:04:17.476629 27564 leader_election.cc:302] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26ca93c0aa7a454eabac2f0ba43990b9, 97cffabceb2c4ccd88bdc03e26bb795c; no voters:
I20260926 15:04:17.476769 27698 raft_consensus.cc:2781] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:04:17.476840 27698 raft_consensus.cc:482] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:04:17.476866 27698 raft_consensus.cc:3037] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:17.477571 27698 raft_consensus.cc:504] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ad1228d2804836b38784b51ca8b9af" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45591 } } peers { permanent_uuid: "26ca93c0aa7a454eabac2f0ba43990b9" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34783 } } peers { permanent_uuid: "97cffabceb2c4ccd88bdc03e26bb795c" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33183 } }
I20260926 15:04:17.477780 27698 leader_election.cc:288] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [CANDIDATE]: Term 1 election: Requested vote from peers a2ad1228d2804836b38784b51ca8b9af (127.8.102.2:45591), 97cffabceb2c4ccd88bdc03e26bb795c (127.8.102.1:33183)
I20260926 15:04:17.477854 27369 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e546146fab4e2f9035c4466e81bf5f" candidate_uuid: "26ca93c0aa7a454eabac2f0ba43990b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97cffabceb2c4ccd88bdc03e26bb795c"
I20260926 15:04:17.477919 27500 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e0e546146fab4e2f9035c4466e81bf5f" candidate_uuid: "26ca93c0aa7a454eabac2f0ba43990b9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2ad1228d2804836b38784b51ca8b9af"
I20260926 15:04:17.477941 27369 raft_consensus.cc:3037] T e0e546146fab4e2f9035c4466e81bf5f P 97cffabceb2c4ccd88bdc03e26bb795c [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:17.477998 27500 raft_consensus.cc:3037] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:17.478787 27369 raft_consensus.cc:2454] T e0e546146fab4e2f9035c4466e81bf5f P 97cffabceb2c4ccd88bdc03e26bb795c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26ca93c0aa7a454eabac2f0ba43990b9 in term 1.
I20260926 15:04:17.478807 27500 raft_consensus.cc:2454] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 26ca93c0aa7a454eabac2f0ba43990b9 in term 1.
I20260926 15:04:17.478997 27564 leader_election.cc:302] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 26ca93c0aa7a454eabac2f0ba43990b9, 97cffabceb2c4ccd88bdc03e26bb795c, a2ad1228d2804836b38784b51ca8b9af; no voters:
I20260926 15:04:17.479166 27698 raft_consensus.cc:2781] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:17.479379 27698 raft_consensus.cc:686] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [term 1 LEADER]: Becoming Leader. State: Replica: 26ca93c0aa7a454eabac2f0ba43990b9, State: Running, Role: LEADER
I20260926 15:04:17.479480 27698 consensus_queue.cc:237] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ad1228d2804836b38784b51ca8b9af" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45591 } } peers { permanent_uuid: "26ca93c0aa7a454eabac2f0ba43990b9" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34783 } } peers { permanent_uuid: "97cffabceb2c4ccd88bdc03e26bb795c" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33183 } }
I20260926 15:04:17.480324 27226 catalog_manager.cc:5495] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 26ca93c0aa7a454eabac2f0ba43990b9 (127.8.102.3). New cstate: current_term: 1 leader_uuid: "26ca93c0aa7a454eabac2f0ba43990b9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a2ad1228d2804836b38784b51ca8b9af" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26ca93c0aa7a454eabac2f0ba43990b9" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 34783 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97cffabceb2c4ccd88bdc03e26bb795c" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33183 } health_report { overall_health: UNKNOWN } } }
I20260926 15:04:17.492079 27689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:17.489419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38692 (local address 127.8.102.3:34783)
0926 15:04:17.489486 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:17.489488 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:17.489496 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:04:17.489594 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:17.489595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:17.489613 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:17.489676 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:04:17.489840 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.490307 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.491252 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:17.491445 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:17.491510 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:17.491728 (+ 218us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:04:17.491815 (+ 87us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:04:17.491895 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:17.491963 (+ 68us) server_negotiation.cc:299] Negotiation successful
0926 15:04:17.491971 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260926 15:04:17.494119 27369 raft_consensus.cc:1261] T e0e546146fab4e2f9035c4466e81bf5f P 97cffabceb2c4ccd88bdc03e26bb795c [term 1 FOLLOWER]: Refusing update from remote peer 26ca93c0aa7a454eabac2f0ba43990b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:04:17.494140 27500 raft_consensus.cc:1261] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af [term 1 FOLLOWER]: Refusing update from remote peer 26ca93c0aa7a454eabac2f0ba43990b9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:04:17.494554 27701 consensus_queue.cc:1035] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2ad1228d2804836b38784b51ca8b9af" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 45591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:04:17.494719 27698 consensus_queue.cc:1035] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97cffabceb2c4ccd88bdc03e26bb795c" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 33183 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:04:17.497776 27704 mvcc.cc:205] Tried to move back new op lower bound from 7333621995492544512 to 7333621995436752896. Current Snapshot: MvccSnapshot[applied={T|T < 7333621995492544512}]
I20260926 15:04:17.498397 27705 mvcc.cc:205] Tried to move back new op lower bound from 7333621995492544512 to 7333621995436752896. Current Snapshot: MvccSnapshot[applied={T|T < 7333621995492544512}]
I20260926 15:04:17.500844 27227 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:04:17.500959 27227 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55364:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:04:17.501844 27227 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c5b7154c5fb34b658bef4e4964df247f: Sending DeleteTablet for 3 replicas of tablet e0e546146fab4e2f9035c4466e81bf5f
I20260926 15:04:17.502259 27611 tablet_service.cc:1514] Processing DeleteTablet for tablet e0e546146fab4e2f9035c4466e81bf5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:04:17 UTC) from {username='slave'} at 127.0.0.1:38680
I20260926 15:04:17.502256 27480 tablet_service.cc:1514] Processing DeleteTablet for tablet e0e546146fab4e2f9035c4466e81bf5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:04:17 UTC) from {username='slave'} at 127.0.0.1:48840
I20260926 15:04:17.502584 27713 tablet_replica.cc:330] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9: stopping tablet replica
I20260926 15:04:17.502562 27349 tablet_service.cc:1514] Processing DeleteTablet for tablet e0e546146fab4e2f9035c4466e81bf5f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:04:17 UTC) from {username='slave'} at 127.0.0.1:35430
I20260926 15:04:17.502751 27713 raft_consensus.cc:2229] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:04:17.502844 27715 tablet_replica.cc:330] T e0e546146fab4e2f9035c4466e81bf5f P 97cffabceb2c4ccd88bdc03e26bb795c: stopping tablet replica
I20260926 15:04:17.502913 27713 raft_consensus.cc:2258] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:04:17.502967 27715 raft_consensus.cc:2229] T e0e546146fab4e2f9035c4466e81bf5f P 97cffabceb2c4ccd88bdc03e26bb795c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:04:17.503041 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 27287
I20260926 15:04:17.503681 27714 tablet_replica.cc:330] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af: stopping tablet replica
I20260926 15:04:17.503801 27714 raft_consensus.cc:2229] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:04:17.503932 27714 raft_consensus.cc:2258] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:04:17.504501 27714 ts_tablet_manager.cc:1905] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:04:17.505179 27713 ts_tablet_manager.cc:1905] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:04:17.506583 27713 ts_tablet_manager.cc:1918] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:04:17.506711 27713 log.cc:1196] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/e0e546146fab4e2f9035c4466e81bf5f
I20260926 15:04:17.506740 27714 ts_tablet_manager.cc:1918] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:04:17.506826 27714 log.cc:1196] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/e0e546146fab4e2f9035c4466e81bf5f
I20260926 15:04:17.506989 27713 ts_tablet_manager.cc:1939] T e0e546146fab4e2f9035c4466e81bf5f P 26ca93c0aa7a454eabac2f0ba43990b9: Deleting consensus metadata
I20260926 15:04:17.507122 27714 ts_tablet_manager.cc:1939] T e0e546146fab4e2f9035c4466e81bf5f P a2ad1228d2804836b38784b51ca8b9af: Deleting consensus metadata
I20260926 15:04:17.507464 27212 catalog_manager.cc:4841] TS 26ca93c0aa7a454eabac2f0ba43990b9 (127.8.102.3:34783): tablet e0e546146fab4e2f9035c4466e81bf5f (table test-table [id=82fd7ed4ad5b4559b7318d1130a9781a]) successfully deleted
I20260926 15:04:17.507786 27213 catalog_manager.cc:4841] TS a2ad1228d2804836b38784b51ca8b9af (127.8.102.2:45591): tablet e0e546146fab4e2f9035c4466e81bf5f (table test-table [id=82fd7ed4ad5b4559b7318d1130a9781a]) successfully deleted
W20260926 15:04:17.509943 27211 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.102.1:33183 (error 108)
I20260926 15:04:17.510179 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 27418
I20260926 15:04:17.512156 27687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:17.511986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:33183 (local address 127.0.0.1:35434)
0926 15:04:17.512063 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:17.512110 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.1:33183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260926 15:04:17.512295 27211 catalog_manager.cc:4584] TS 97cffabceb2c4ccd88bdc03e26bb795c (127.8.102.1:33183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e0e546146fab4e2f9035c4466e81bf5f: Network error: Client connection negotiation failed: client connection to 127.8.102.1:33183: connect: Connection refused (error 111)
I20260926 15:04:17.515548 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 27549
I20260926 15:04:17.520473 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 27196
2026-09-26T15:04:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (680 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:17 dist-test-slave-jnhs krb5kdc[27719](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:17 dist-test-slave-jnhs krb5kdc[27719](info): set up 2 sockets
Sep 26 15:04:17 dist-test-slave-jnhs krb5kdc[27719](info): commencing operation
krb5kdc: starting...
W20260926 15:04:18.602789 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.062s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:18 dist-test-slave-jnhs krb5kdc[27719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:04:18.735270 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:44303
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:44303
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:18.819075 27732 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:18.819485 27732 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:18.819533 27732 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:18.821353 27732 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:18.821424 27732 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:18.821439 27732 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:18.821452 27732 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:18.821465 27732 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:18.823222 27732 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:44303
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:44303
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27732
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:18.823455 27732 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:18.823639 27732 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:18.828756 27737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:18.828783 27732 server_base.cc:1034] running on GCE node
W20260926 15:04:18.828756 27740 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:18.828735 27738 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:18.829274 27732 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:18.829586 27732 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:18.830750 27732 hybrid_clock.cc:648] HybridClock initialized: now 1790435058830727 us; error 55 us; skew 500 ppm
Sep 26 15:04:18 dist-test-slave-jnhs krb5kdc[27719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435058, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:18.833413 27732 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:04:18.833930 27732 webserver.cc:460] Webserver started at http://127.8.102.62:36497/ using document root <none> and password file <none>
I20260926 15:04:18.834172 27732 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:18.834242 27732 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:18.834415 27732 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:18.835592 27732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "f680e099a23f480287124e3ae5d7dd79"
format_stamp: "Formatted at 2026-09-26 15:04:18 on dist-test-slave-jnhs"
server_key: "d3ed9c8b4595f00d9fa27c8d9997735f"
server_key_iv: "5cfe3dfe46b4a636c32e90e1ae4d927c"
server_key_version: "encryptionkey@0"
I20260926 15:04:18.835988 27732 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "f680e099a23f480287124e3ae5d7dd79"
format_stamp: "Formatted at 2026-09-26 15:04:18 on dist-test-slave-jnhs"
server_key: "d3ed9c8b4595f00d9fa27c8d9997735f"
server_key_iv: "5cfe3dfe46b4a636c32e90e1ae4d927c"
server_key_version: "encryptionkey@0"
I20260926 15:04:18.837459 27732 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:04:18.838250 27747 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:18.838445 27732 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:18.838549 27732 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "f680e099a23f480287124e3ae5d7dd79"
format_stamp: "Formatted at 2026-09-26 15:04:18 on dist-test-slave-jnhs"
server_key: "d3ed9c8b4595f00d9fa27c8d9997735f"
server_key_iv: "5cfe3dfe46b4a636c32e90e1ae4d927c"
server_key_version: "encryptionkey@0"
I20260926 15:04:18.838634 27732 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:18.850136 27732 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:18.852411 27732 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:18.852594 27732 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:18.857095 27732 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:44303
I20260926 15:04:18.857127 27799 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:44303 every 8 connection(s)
I20260926 15:04:18.857517 27732 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:18.858479 27800 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:18.861142 27800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79: Bootstrap starting.
I20260926 15:04:18.861166 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 27732
I20260926 15:04:18.861301 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:18.861487 8600 external_mini_cluster.cc:1428] Setting key f9c7b6a16fbfda27b58856a7b3bd5975
I20260926 15:04:18.861971 27800 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:18.862378 27800 log.cc:824] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:18.863241 27800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79: No bootstrap required, opened a new log
I20260926 15:04:18.865867 27800 raft_consensus.cc:348] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f680e099a23f480287124e3ae5d7dd79" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44303 } }
I20260926 15:04:18.866073 27800 raft_consensus.cc:374] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:18.866108 27800 raft_consensus.cc:729] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f680e099a23f480287124e3ae5d7dd79, State: Initialized, Role: FOLLOWER
I20260926 15:04:18.866247 27800 consensus_queue.cc:260] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f680e099a23f480287124e3ae5d7dd79" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44303 } }
I20260926 15:04:18.866343 27800 raft_consensus.cc:388] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:18.866389 27800 raft_consensus.cc:482] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:18.866439 27800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:18.867281 27800 raft_consensus.cc:504] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f680e099a23f480287124e3ae5d7dd79" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44303 } }
I20260926 15:04:18.867412 27800 leader_election.cc:302] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f680e099a23f480287124e3ae5d7dd79; no voters:
I20260926 15:04:18.867625 27800 leader_election.cc:288] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:18.867682 27805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:18.867928 27800 sys_catalog.cc:564] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:18.867950 27805 raft_consensus.cc:686] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79 [term 1 LEADER]: Becoming Leader. State: Replica: f680e099a23f480287124e3ae5d7dd79, State: Running, Role: LEADER
I20260926 15:04:18.868058 27805 consensus_queue.cc:237] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f680e099a23f480287124e3ae5d7dd79" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44303 } }
I20260926 15:04:18.868387 27806 sys_catalog.cc:455] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f680e099a23f480287124e3ae5d7dd79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f680e099a23f480287124e3ae5d7dd79" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44303 } } }
I20260926 15:04:18.868467 27806 sys_catalog.cc:458] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:18.868711 27807 sys_catalog.cc:455] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f680e099a23f480287124e3ae5d7dd79. Latest consensus state: current_term: 1 leader_uuid: "f680e099a23f480287124e3ae5d7dd79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f680e099a23f480287124e3ae5d7dd79" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44303 } } }
I20260926 15:04:18.869129 27811 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:18.868827 27807 sys_catalog.cc:458] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:18.869704 27811 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 15:04:18 dist-test-slave-jnhs krb5kdc[27719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435058, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:04:18.871719 27811 catalog_manager.cc:1347] Generated new cluster ID: 3cb29132ade54d2ca58aad2ff589443a
I20260926 15:04:18.871778 27811 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:18.874540 27803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:18.862166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40798 (local address 127.8.102.62:44303)
0926 15:04:18.862360 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:18.862370 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:18.862413 (+ 43us) server_negotiation.cc:407] Connection header received
0926 15:04:18.863097 (+ 684us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:18.863106 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:18.863162 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:18.863290 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:18.864214 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:18.865157 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:18.866369 (+ 1212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:18.866957 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:18.867052 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:18.870614 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:18.870620 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:18.870620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:18.870660 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:18.872735 (+ 2075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:18.873178 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:18.873181 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:18.873182 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:18.873286 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:18.873527 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:18.873528 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:18.873529 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:18.873803 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:18.873994 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:18.874132 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 15:04:18.874289 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":101,"threads_started":1}
I20260926 15:04:18.897440 27811 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:18.898160 27811 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:18.912335 27811 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79: Generated new TSK 0
I20260926 15:04:18.912606 27811 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.8.102.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
Entry for principal kudu/127.8.102.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab.
I20260926 15:04:18.950806 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44303
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:19.035732 27826 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:19.035955 27826 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:19.036002 27826 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:19.037200 27826 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:19.037240 27826 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:19.037268 27826 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:04:19.038834 27826 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.1.keytab
--principal=kudu/127.8.102.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27826
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:19.039049 27826 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:19.039242 27826 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:19.044579 27831 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:19.044584 27832 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:19.044579 27834 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:19.044884 27826 server_base.cc:1034] running on GCE node
I20260926 15:04:19.045151 27826 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:19.045428 27826 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:19.046685 27826 hybrid_clock.cc:648] HybridClock initialized: now 1790435059046622 us; error 36 us; skew 500 ppm
Sep 26 15:04:19 dist-test-slave-jnhs krb5kdc[27719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435059, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:19.049372 27826 init.cc:377] Logged in from keytab as kudu/127.8.102.1@KRBTEST.COM (short username kudu)
I20260926 15:04:19.049912 27826 webserver.cc:460] Webserver started at http://127.8.102.1:34541/ using document root <none> and password file <none>
I20260926 15:04:19.050191 27826 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:19.050266 27826 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:19.050431 27826 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:19.051676 27826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "989d2f289c3a4604a94c57d635989c94"
format_stamp: "Formatted at 2026-09-26 15:04:19 on dist-test-slave-jnhs"
server_key: "f671944d2abad66614b399f986de28f8"
server_key_iv: "e4b931dc33c232e29ea777a6e1444867"
server_key_version: "encryptionkey@0"
I20260926 15:04:19.052062 27826 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "989d2f289c3a4604a94c57d635989c94"
format_stamp: "Formatted at 2026-09-26 15:04:19 on dist-test-slave-jnhs"
server_key: "f671944d2abad66614b399f986de28f8"
server_key_iv: "e4b931dc33c232e29ea777a6e1444867"
server_key_version: "encryptionkey@0"
I20260926 15:04:19.053503 27826 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 15:04:19.054394 27841 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:19.054703 27826 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:19.054816 27826 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "989d2f289c3a4604a94c57d635989c94"
format_stamp: "Formatted at 2026-09-26 15:04:19 on dist-test-slave-jnhs"
server_key: "f671944d2abad66614b399f986de28f8"
server_key_iv: "e4b931dc33c232e29ea777a6e1444867"
server_key_version: "encryptionkey@0"
I20260926 15:04:19.054913 27826 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:19.084708 27826 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:19.087198 27826 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:19.087356 27826 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:19.087616 27826 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:04:19.087950 27826 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:04:19.087982 27826 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:19.088035 27826 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:04:19.088063 27826 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:19.093788 27826 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:36059
I20260926 15:04:19.093940 27954 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:36059 every 8 connection(s)
I20260926 15:04:19.094223 27826 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:04:19.096184 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 27826
I20260926 15:04:19.096311 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:04:19.096603 8600 external_mini_cluster.cc:1428] Setting key dc5bbe670090fc4c3e99b3d3acf402d2
Sep 26 15:04:19 dist-test-slave-jnhs krb5kdc[27719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435059, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.1@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.102.2@KRBTEST.COM; defaulting to no policy
I20260926 15:04:19.106127 27803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:19.095137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:50589 (local address 127.8.102.62:44303)
0926 15:04:19.095232 (+ 95us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:19.095234 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:19.095487 (+ 253us) server_negotiation.cc:407] Connection header received
0926 15:04:19.096039 (+ 552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:19.096040 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:19.096053 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:19.096108 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:19.097335 (+ 1227us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.097864 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.099389 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.099742 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.099826 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:19.102578 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:19.102580 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:19.102580 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:19.102589 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:19.104328 (+ 1739us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:19.104988 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:19.104989 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:19.104990 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:19.105058 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:19.105411 (+ 353us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:19.105412 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:19.105412 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:19.105538 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:19.105634 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:19.105916 (+ 282us) server_negotiation.cc:299] Negotiation successful
0926 15:04:19.106024 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260926 15:04:19.106377 27957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:19.095115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44303 (local address 127.8.102.1:50589)
0926 15:04:19.095319 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:19.095367 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:19.095877 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:19.096216 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:19.096221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:19.097039 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:19.097256 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:19.097261 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.097973 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.097974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:19.099292 (+ 1318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:19.099295 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.099859 (+ 564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.099860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:19.099983 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:19.100617 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:19.100628 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:19.102370 (+ 1742us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:19.104449 (+ 2079us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:19.104459 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:19.104462 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:19.104860 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:19.105147 (+ 287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:19.105148 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:19.105148 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:19.105317 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:19.105626 (+ 309us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:19.105633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:19.105779 (+ 146us) client_negotiation.cc:769] Sending connection context
0926 15:04:19.105895 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 15:04:19.106075 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":117,"threads_started":1}
I20260926 15:04:19.107048 27955 heartbeater.cc:347] Connected to a master server at 127.8.102.62:44303
I20260926 15:04:19.107160 27955 heartbeater.cc:464] Registering TS with master...
I20260926 15:04:19.107400 27955 heartbeater.cc:511] Master 127.8.102.62:44303 requested a full tablet report, sending...
I20260926 15:04:19.107967 27764 ts_manager.cc:194] Registered new tserver with Master: 989d2f289c3a4604a94c57d635989c94 (127.8.102.1:36059)
I20260926 15:04:19.108791 27764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.1@KRBTEST.COM'} at 127.8.102.1:50589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
Entry for principal kudu/127.8.102.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab.
I20260926 15:04:19.126735 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44303
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:19.213020 27960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:19.213240 27960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:19.213267 27960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:19.214525 27960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:19.214581 27960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:19.214612 27960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:04:19.216285 27960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.2.keytab
--principal=kudu/127.8.102.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27960
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:19.216560 27960 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:19.216812 27960 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:19.222126 27968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:19.222126 27966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:19.222257 27965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:19.222297 27960 server_base.cc:1034] running on GCE node
I20260926 15:04:19.222777 27960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:19.223091 27960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:19.224270 27960 hybrid_clock.cc:648] HybridClock initialized: now 1790435059224256 us; error 38 us; skew 500 ppm
Sep 26 15:04:19 dist-test-slave-jnhs krb5kdc[27719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435059, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:19.227195 27960 init.cc:377] Logged in from keytab as kudu/127.8.102.2@KRBTEST.COM (short username kudu)
I20260926 15:04:19.227617 27960 webserver.cc:460] Webserver started at http://127.8.102.2:40285/ using document root <none> and password file <none>
I20260926 15:04:19.227885 27960 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:19.227936 27960 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:19.228106 27960 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:19.229214 27960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "dacea321aaf647318b088b32d465fd0a"
format_stamp: "Formatted at 2026-09-26 15:04:19 on dist-test-slave-jnhs"
server_key: "d836123402f88f7fd68c0d60d2423b14"
server_key_iv: "06370734272d8162c17ac9172aed1501"
server_key_version: "encryptionkey@0"
I20260926 15:04:19.229585 27960 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "dacea321aaf647318b088b32d465fd0a"
format_stamp: "Formatted at 2026-09-26 15:04:19 on dist-test-slave-jnhs"
server_key: "d836123402f88f7fd68c0d60d2423b14"
server_key_iv: "06370734272d8162c17ac9172aed1501"
server_key_version: "encryptionkey@0"
I20260926 15:04:19.230983 27960 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 15:04:19.231848 27975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:19.232017 27960 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:04:19.232112 27960 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "dacea321aaf647318b088b32d465fd0a"
format_stamp: "Formatted at 2026-09-26 15:04:19 on dist-test-slave-jnhs"
server_key: "d836123402f88f7fd68c0d60d2423b14"
server_key_iv: "06370734272d8162c17ac9172aed1501"
server_key_version: "encryptionkey@0"
I20260926 15:04:19.232195 27960 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:19.244829 27960 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:19.247371 27960 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:19.247548 27960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:19.247843 27960 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:04:19.248196 27960 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:04:19.248253 27960 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:19.248298 27960 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:04:19.248327 27960 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:19.255173 27960 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:41321
I20260926 15:04:19.255265 28088 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:41321 every 8 connection(s)
I20260926 15:04:19.255623 27960 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:04:19.262328 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 27960
I20260926 15:04:19.262434 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:04:19.262677 8600 external_mini_cluster.cc:1428] Setting key f21c381e28d2a555fca6274af868113e
Sep 26 15:04:19 dist-test-slave-jnhs krb5kdc[27719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435059, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.2@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:04:19.267783 27803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:19.256561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:38569 (local address 127.8.102.62:44303)
0926 15:04:19.256654 (+ 93us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:19.256657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:19.256994 (+ 337us) server_negotiation.cc:407] Connection header received
0926 15:04:19.258128 (+ 1134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:19.258129 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:19.258143 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:19.258206 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:19.258992 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.259536 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.261185 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.261434 (+ 249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.261504 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:19.264153 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:19.264155 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:19.264155 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:19.264165 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:19.266037 (+ 1872us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:19.266545 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:19.266547 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:19.266547 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:19.266624 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:19.266941 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:19.266942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:19.266943 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:19.267123 (+ 180us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:19.267204 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:19.267541 (+ 337us) server_negotiation.cc:299] Negotiation successful
0926 15:04:19.267658 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260926 15:04:19.267865 28091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:19.256430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44303 (local address 127.8.102.2:38569)
0926 15:04:19.256747 (+ 317us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:19.256798 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:19.257994 (+ 1196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:19.258265 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:19.258269 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:19.258572 (+ 303us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:19.258869 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:19.258875 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.259683 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.259684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:19.261102 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:19.261106 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.261533 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.261534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:19.261656 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:19.262257 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:19.262266 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:19.264048 (+ 1782us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:19.266166 (+ 2118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:19.266173 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:19.266175 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:19.266424 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:19.266737 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:19.266738 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:19.266738 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:19.266863 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:19.267222 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:19.267228 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:19.267365 (+ 137us) client_negotiation.cc:769] Sending connection context
0926 15:04:19.267475 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 15:04:19.267620 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"thread_start_us":115,"threads_started":1}
I20260926 15:04:19.268419 28089 heartbeater.cc:347] Connected to a master server at 127.8.102.62:44303
I20260926 15:04:19.268549 28089 heartbeater.cc:464] Registering TS with master...
I20260926 15:04:19.268797 28089 heartbeater.cc:511] Master 127.8.102.62:44303 requested a full tablet report, sending...
I20260926 15:04:19.269330 27764 ts_manager.cc:194] Registered new tserver with Master: dacea321aaf647318b088b32d465fd0a (127.8.102.2:41321)
I20260926 15:04:19.269744 27764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.2@KRBTEST.COM'} at 127.8.102.2:38569
WARNING: no policy specified for kudu/127.8.102.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
Entry for principal kudu/127.8.102.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab.
I20260926 15:04:19.294580 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44303
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:34337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:19.381574 28094 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:19.381788 28094 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:19.381812 28094 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:19.383158 28094 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:19.383208 28094 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:19.383239 28094 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:04:19.384672 28094 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:34337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.3.keytab
--principal=kudu/127.8.102.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:44303
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28094
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:19.384874 28094 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:19.385066 28094 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:19.390355 28099 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:19.390350 28100 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:19.390350 28102 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:19.390481 28094 server_base.cc:1034] running on GCE node
I20260926 15:04:19.390846 28094 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:19.391144 28094 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:19.392319 28094 hybrid_clock.cc:648] HybridClock initialized: now 1790435059392299 us; error 38 us; skew 500 ppm
Sep 26 15:04:19 dist-test-slave-jnhs krb5kdc[27719](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435059, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:19.395008 28094 init.cc:377] Logged in from keytab as kudu/127.8.102.3@KRBTEST.COM (short username kudu)
I20260926 15:04:19.395511 28094 webserver.cc:460] Webserver started at http://127.8.102.3:39171/ using document root <none> and password file <none>
I20260926 15:04:19.395733 28094 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:19.395778 28094 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:19.395870 28094 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:19.396942 28094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "7b19d303dec4492ea09fa238055aa4c4"
format_stamp: "Formatted at 2026-09-26 15:04:19 on dist-test-slave-jnhs"
server_key: "5c2ebc9adde9f99ee25f3d9e4370617d"
server_key_iv: "63d9c0b7b88358f6904331c37fe9b8cc"
server_key_version: "encryptionkey@0"
I20260926 15:04:19.397249 28094 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "7b19d303dec4492ea09fa238055aa4c4"
format_stamp: "Formatted at 2026-09-26 15:04:19 on dist-test-slave-jnhs"
server_key: "5c2ebc9adde9f99ee25f3d9e4370617d"
server_key_iv: "63d9c0b7b88358f6904331c37fe9b8cc"
server_key_version: "encryptionkey@0"
I20260926 15:04:19.398555 28094 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:04:19.399381 28109 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:19.399578 28094 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:19.399662 28094 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "7b19d303dec4492ea09fa238055aa4c4"
format_stamp: "Formatted at 2026-09-26 15:04:19 on dist-test-slave-jnhs"
server_key: "5c2ebc9adde9f99ee25f3d9e4370617d"
server_key_iv: "63d9c0b7b88358f6904331c37fe9b8cc"
server_key_version: "encryptionkey@0"
I20260926 15:04:19.399735 28094 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:19.414270 28094 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:19.416730 28094 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:19.416883 28094 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:19.417130 28094 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:04:19.417448 28094 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:04:19.417482 28094 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:19.417505 28094 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:04:19.417551 28094 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:19.423966 28094 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:36797
I20260926 15:04:19.424023 28222 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:36797 every 8 connection(s)
I20260926 15:04:19.424407 28094 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:04:19.430403 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 28094
I20260926 15:04:19.430546 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:04:19.430783 8600 external_mini_cluster.cc:1428] Setting key 760496b0f7c3d3b4c87517b4695a4b57
Sep 26 15:04:19 dist-test-slave-jnhs krb5kdc[27719](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435059, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.3@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:04:19.435273 27803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:19.425217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:60745 (local address 127.8.102.62:44303)
0926 15:04:19.425282 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:19.425284 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:19.425474 (+ 190us) server_negotiation.cc:407] Connection header received
0926 15:04:19.425996 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:19.425997 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:19.426008 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:19.426072 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:19.427270 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.427863 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.429027 (+ 1164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.429259 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.429337 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:19.432154 (+ 2817us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:19.432155 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:19.432156 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:19.432248 (+ 92us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:19.433724 (+ 1476us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:19.434140 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:19.434141 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:19.434141 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:19.434232 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:19.434532 (+ 300us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:19.434532 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:19.434532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:19.434693 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:19.434757 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:19.435051 (+ 294us) server_negotiation.cc:299] Negotiation successful
0926 15:04:19.435161 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 15:04:19.435568 28225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:19.425210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44303 (local address 127.8.102.3:60745)
0926 15:04:19.425365 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:19.425402 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:19.425881 (+ 479us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:19.426177 (+ 296us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:19.426180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:19.426933 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:19.427157 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:19.427162 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.428037 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.428038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:19.428922 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:19.428925 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.429381 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.429382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:19.429536 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:19.430238 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:19.430248 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:19.432067 (+ 1819us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:19.433815 (+ 1748us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:19.433822 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:19.433824 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:19.434069 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:19.434335 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:19.434336 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:19.434336 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:19.434474 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:19.434773 (+ 299us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:19.434780 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:19.434928 (+ 148us) client_negotiation.cc:769] Sending connection context
0926 15:04:19.435030 (+ 102us) client_negotiation.cc:240] Negotiation successful
0926 15:04:19.435280 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":75,"threads_started":1}
I20260926 15:04:19.436067 28223 heartbeater.cc:347] Connected to a master server at 127.8.102.62:44303
I20260926 15:04:19.436151 28223 heartbeater.cc:464] Registering TS with master...
I20260926 15:04:19.436316 28223 heartbeater.cc:511] Master 127.8.102.62:44303 requested a full tablet report, sending...
I20260926 15:04:19.436654 27764 ts_manager.cc:194] Registered new tserver with Master: 7b19d303dec4492ea09fa238055aa4c4 (127.8.102.3:36797)
I20260926 15:04:19.437071 27764 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.8.102.3@KRBTEST.COM'} at 127.8.102.3:60745
I20260926 15:04:19.443276 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:04:19.451889 27803 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:19.444747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40804 (local address 127.8.102.62:44303)
0926 15:04:19.444818 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:19.444820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:19.444928 (+ 108us) server_negotiation.cc:407] Connection header received
0926 15:04:19.444970 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:19.444970 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:19.444981 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:19.445039 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:19.445645 (+ 606us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.446195 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.447066 (+ 871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.447298 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.447372 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:19.448827 (+ 1455us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:19.448828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:19.448829 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:19.448836 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:19.450503 (+ 1667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:19.450954 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:19.450955 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:19.450956 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:19.451016 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:19.451308 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:19.451309 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:19.451309 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:19.451480 (+ 171us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:19.451564 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:19.451744 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 15:04:19.451818 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 15:04:19.454033 27764 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:04:19.456281 27764 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:04:19.464305 28239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:19.459947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43850 (local address 127.8.102.1:36059)
0926 15:04:19.460149 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:19.460153 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:19.460163 (+ 10us) server_negotiation.cc:407] Connection header received
0926 15:04:19.460203 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:19.460205 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:19.460219 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:19.460447 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:19.460539 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.461617 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.463322 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.463863 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.463967 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:19.464072 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:19.464170 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 15:04:19.464212 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":98,"threads_started":1}
I20260926 15:04:19.464301 28235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:19.459384 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:36059 (local address 127.0.0.1:43850)
0926 15:04:19.459548 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:19.459559 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:19.459615 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:19.460322 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:19.460324 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:19.460326 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:19.460375 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:19.460376 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.461713 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.461714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:19.463223 (+ 1509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:19.463225 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.463970 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.463971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:19.464064 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:19.464080 (+ 16us) client_negotiation.cc:769] Sending connection context
0926 15:04:19.464155 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 15:04:19.464203 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"mutex_wait_us":27,"thread_start_us":118,"threads_started":1}
I20260926 15:04:19.464485 28238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:19.459783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47114 (local address 127.8.102.2:41321)
0926 15:04:19.459950 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:19.459953 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:19.460218 (+ 265us) server_negotiation.cc:407] Connection header received
0926 15:04:19.460370 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:19.460372 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:19.460388 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:19.460444 (+ 56us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:19.460650 (+ 206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.461398 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.463305 (+ 1907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.464032 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.464107 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:19.464169 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:19.464355 (+ 186us) server_negotiation.cc:299] Negotiation successful
0926 15:04:19.464398 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133,"thread_start_us":73,"threads_started":1}
I20260926 15:04:19.464751 28237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:19.459666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:41321 (local address 127.0.0.1:47114)
0926 15:04:19.460154 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:19.460164 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:19.460274 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:19.460537 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:19.460538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:19.460540 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:19.460581 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:19.460582 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.461517 (+ 935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.461518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:19.463195 (+ 1677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:19.463198 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.464203 (+ 1005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.464204 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:19.464261 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:19.464265 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:04:19.464571 (+ 306us) client_negotiation.cc:240] Negotiation successful
0926 15:04:19.464621 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"mutex_wait_us":6,"thread_start_us":76,"threads_started":1}
I20260926 15:04:19.464732 28234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:19.459257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:36797 (local address 127.0.0.1:41590)
0926 15:04:19.459410 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:19.459456 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:19.459529 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:19.460034 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:19.460037 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:19.460040 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:19.460126 (+ 86us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:19.460129 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.461753 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.461754 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:19.463005 (+ 1251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:19.463009 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.464019 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.464020 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:19.464092 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:19.464100 (+ 8us) client_negotiation.cc:769] Sending connection context
0926 15:04:19.464131 (+ 31us) client_negotiation.cc:240] Negotiation successful
0926 15:04:19.464206 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"mutex_wait_us":19,"thread_start_us":101,"threads_started":1}
I20260926 15:04:19.464891 28236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:19.459479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41590 (local address 127.8.102.3:36797)
0926 15:04:19.459814 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:19.459824 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:19.459833 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:04:19.459893 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:19.459895 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:19.459913 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:19.460003 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:19.460324 (+ 321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.461595 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.463135 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.463902 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.464262 (+ 360us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:19.464333 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:19.464357 (+ 24us) server_negotiation.cc:299] Negotiation successful
0926 15:04:19.464398 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":94,"threads_started":1}
I20260926 15:04:19.465406 27889 tablet_service.cc:1467] Processing CreateTablet for tablet f04241ad4f2a4cba809899ec5670b038 (DEFAULT_TABLE table=test-table [id=ec32af4db974434aa58107f818157943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:04:19.465541 28157 tablet_service.cc:1467] Processing CreateTablet for tablet f04241ad4f2a4cba809899ec5670b038 (DEFAULT_TABLE table=test-table [id=ec32af4db974434aa58107f818157943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:04:19.465721 27889 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f04241ad4f2a4cba809899ec5670b038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:19.465915 28157 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f04241ad4f2a4cba809899ec5670b038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:19.467478 28023 tablet_service.cc:1467] Processing CreateTablet for tablet f04241ad4f2a4cba809899ec5670b038 (DEFAULT_TABLE table=test-table [id=ec32af4db974434aa58107f818157943]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:04:19.467847 28023 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f04241ad4f2a4cba809899ec5670b038. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:19.469152 28241 tablet_bootstrap.cc:492] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4: Bootstrap starting.
I20260926 15:04:19.469548 28240 tablet_bootstrap.cc:492] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94: Bootstrap starting.
I20260926 15:04:19.469956 28241 tablet_bootstrap.cc:654] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:19.470227 28240 tablet_bootstrap.cc:654] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:19.470613 28241 log.cc:824] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:19.470949 28240 log.cc:824] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:19.471567 28244 tablet_bootstrap.cc:492] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a: Bootstrap starting.
I20260926 15:04:19.471927 28241 tablet_bootstrap.cc:492] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4: No bootstrap required, opened a new log
I20260926 15:04:19.472011 28241 ts_tablet_manager.cc:1397] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.001s
I20260926 15:04:19.472254 28244 tablet_bootstrap.cc:654] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:19.472321 28240 tablet_bootstrap.cc:492] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94: No bootstrap required, opened a new log
I20260926 15:04:19.472389 28240 ts_tablet_manager.cc:1397] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94: Time spent bootstrapping tablet: real 0.003s user 0.002s sys 0.000s
I20260926 15:04:19.472863 28244 log.cc:824] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:19.473598 28241 raft_consensus.cc:348] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b19d303dec4492ea09fa238055aa4c4" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36797 } } peers { permanent_uuid: "989d2f289c3a4604a94c57d635989c94" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36059 } } peers { permanent_uuid: "dacea321aaf647318b088b32d465fd0a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41321 } }
I20260926 15:04:19.473734 28241 raft_consensus.cc:374] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:19.473755 28241 raft_consensus.cc:729] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b19d303dec4492ea09fa238055aa4c4, State: Initialized, Role: FOLLOWER
I20260926 15:04:19.473747 28244 tablet_bootstrap.cc:492] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a: No bootstrap required, opened a new log
I20260926 15:04:19.473814 28244 ts_tablet_manager.cc:1397] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:04:19.473887 28241 consensus_queue.cc:260] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b19d303dec4492ea09fa238055aa4c4" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36797 } } peers { permanent_uuid: "989d2f289c3a4604a94c57d635989c94" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36059 } } peers { permanent_uuid: "dacea321aaf647318b088b32d465fd0a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41321 } }
I20260926 15:04:19.474182 28223 heartbeater.cc:503] Master 127.8.102.62:44303 was elected leader, sending a full tablet report...
I20260926 15:04:19.474189 28241 ts_tablet_manager.cc:1428] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260926 15:04:19.474596 28240 raft_consensus.cc:348] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b19d303dec4492ea09fa238055aa4c4" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36797 } } peers { permanent_uuid: "989d2f289c3a4604a94c57d635989c94" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36059 } } peers { permanent_uuid: "dacea321aaf647318b088b32d465fd0a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41321 } }
I20260926 15:04:19.474809 28240 raft_consensus.cc:374] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:19.474848 28240 raft_consensus.cc:729] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 989d2f289c3a4604a94c57d635989c94, State: Initialized, Role: FOLLOWER
I20260926 15:04:19.474997 28240 consensus_queue.cc:260] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b19d303dec4492ea09fa238055aa4c4" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36797 } } peers { permanent_uuid: "989d2f289c3a4604a94c57d635989c94" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36059 } } peers { permanent_uuid: "dacea321aaf647318b088b32d465fd0a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41321 } }
I20260926 15:04:19.475400 28240 ts_tablet_manager.cc:1428] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:04:19.475533 27955 heartbeater.cc:503] Master 127.8.102.62:44303 was elected leader, sending a full tablet report...
I20260926 15:04:19.476384 28244 raft_consensus.cc:348] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b19d303dec4492ea09fa238055aa4c4" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36797 } } peers { permanent_uuid: "989d2f289c3a4604a94c57d635989c94" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36059 } } peers { permanent_uuid: "dacea321aaf647318b088b32d465fd0a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41321 } }
I20260926 15:04:19.476601 28244 raft_consensus.cc:374] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:19.476653 28244 raft_consensus.cc:729] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dacea321aaf647318b088b32d465fd0a, State: Initialized, Role: FOLLOWER
I20260926 15:04:19.476773 28244 consensus_queue.cc:260] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b19d303dec4492ea09fa238055aa4c4" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36797 } } peers { permanent_uuid: "989d2f289c3a4604a94c57d635989c94" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36059 } } peers { permanent_uuid: "dacea321aaf647318b088b32d465fd0a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41321 } }
I20260926 15:04:19.477046 28244 ts_tablet_manager.cc:1428] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260926 15:04:19.477090 28089 heartbeater.cc:503] Master 127.8.102.62:44303 was elected leader, sending a full tablet report...
W20260926 15:04:19.506316 28090 tablet.cc:2369] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260926 15:04:19.595069 27956 tablet.cc:2369] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:04:19.645617 28247 raft_consensus.cc:482] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:04:19.645741 28247 raft_consensus.cc:504] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b19d303dec4492ea09fa238055aa4c4" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36797 } } peers { permanent_uuid: "989d2f289c3a4604a94c57d635989c94" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36059 } } peers { permanent_uuid: "dacea321aaf647318b088b32d465fd0a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41321 } }
I20260926 15:04:19.646138 28247 leader_election.cc:288] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b19d303dec4492ea09fa238055aa4c4 (127.8.102.3:36797), dacea321aaf647318b088b32d465fd0a (127.8.102.2:41321)
I20260926 15:04:19.649257 28238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:19.646478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:46921 (local address 127.8.102.2:41321)
0926 15:04:19.646546 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:19.646548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:19.646775 (+ 227us) server_negotiation.cc:407] Connection header received
0926 15:04:19.646806 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:19.646807 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:19.646820 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:19.646898 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:19.647076 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.647588 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.648506 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.648891 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.648947 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:19.648956 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:19.649134 (+ 178us) server_negotiation.cc:299] Negotiation successful
0926 15:04:19.649181 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260926 15:04:19.649298 28236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:19.646439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:44897 (local address 127.8.102.3:36797)
0926 15:04:19.646493 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:19.646496 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:19.646523 (+ 27us) server_negotiation.cc:407] Connection header received
0926 15:04:19.646684 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:19.646685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:19.646698 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:19.646759 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:19.646957 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.647469 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.648538 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.648903 (+ 365us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.648994 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:19.649003 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:19.649188 (+ 185us) server_negotiation.cc:299] Negotiation successful
0926 15:04:19.649225 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 15:04:19.649326 28250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:19.646327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:41321 (local address 127.8.102.1:46921)
0926 15:04:19.646706 (+ 379us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:19.646715 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:19.646762 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:19.646906 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:19.646906 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:19.646908 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:19.647005 (+ 97us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:19.647007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.647673 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.647673 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:19.648410 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:19.648412 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.648975 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.648976 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:19.649057 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:19.649061 (+ 4us) client_negotiation.cc:769] Sending connection context
0926 15:04:19.649120 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 15:04:19.649182 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":366,"mutex_wait_us":2,"thread_start_us":61,"threads_started":1}
I20260926 15:04:19.649494 28249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:19.646327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:36797 (local address 127.8.102.1:44897)
0926 15:04:19.646462 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:19.646476 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:19.646573 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:19.646794 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:19.646795 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:19.646798 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:19.646880 (+ 82us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:19.646882 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.647545 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.647546 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:19.648413 (+ 867us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:19.648416 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.649013 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.649015 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:19.649087 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:19.649090 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:04:19.649165 (+ 75us) client_negotiation.cc:240] Negotiation successful
0926 15:04:19.649208 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"mutex_wait_us":2,"thread_start_us":71,"threads_started":1}
I20260926 15:04:19.649712 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04241ad4f2a4cba809899ec5670b038" candidate_uuid: "989d2f289c3a4604a94c57d635989c94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dacea321aaf647318b088b32d465fd0a" is_pre_election: true
I20260926 15:04:19.649840 28177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04241ad4f2a4cba809899ec5670b038" candidate_uuid: "989d2f289c3a4604a94c57d635989c94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b19d303dec4492ea09fa238055aa4c4" is_pre_election: true
I20260926 15:04:19.649895 28043 raft_consensus.cc:2454] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 989d2f289c3a4604a94c57d635989c94 in term 0.
I20260926 15:04:19.649971 28177 raft_consensus.cc:2454] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 989d2f289c3a4604a94c57d635989c94 in term 0.
I20260926 15:04:19.650149 27845 leader_election.cc:302] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 989d2f289c3a4604a94c57d635989c94, dacea321aaf647318b088b32d465fd0a; no voters:
I20260926 15:04:19.650306 28247 raft_consensus.cc:2781] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:04:19.650393 28247 raft_consensus.cc:482] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:04:19.650444 28247 raft_consensus.cc:3037] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:19.651274 28247 raft_consensus.cc:504] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b19d303dec4492ea09fa238055aa4c4" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36797 } } peers { permanent_uuid: "989d2f289c3a4604a94c57d635989c94" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36059 } } peers { permanent_uuid: "dacea321aaf647318b088b32d465fd0a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41321 } }
I20260926 15:04:19.651443 28247 leader_election.cc:288] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94 [CANDIDATE]: Term 1 election: Requested vote from peers 7b19d303dec4492ea09fa238055aa4c4 (127.8.102.3:36797), dacea321aaf647318b088b32d465fd0a (127.8.102.2:41321)
I20260926 15:04:19.651625 28043 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04241ad4f2a4cba809899ec5670b038" candidate_uuid: "989d2f289c3a4604a94c57d635989c94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dacea321aaf647318b088b32d465fd0a"
I20260926 15:04:19.651671 28177 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f04241ad4f2a4cba809899ec5670b038" candidate_uuid: "989d2f289c3a4604a94c57d635989c94" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b19d303dec4492ea09fa238055aa4c4"
I20260926 15:04:19.651737 28043 raft_consensus.cc:3037] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:19.651749 28177 raft_consensus.cc:3037] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:19.652756 28043 raft_consensus.cc:2454] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 989d2f289c3a4604a94c57d635989c94 in term 1.
I20260926 15:04:19.652756 28177 raft_consensus.cc:2454] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 989d2f289c3a4604a94c57d635989c94 in term 1.
I20260926 15:04:19.653012 27845 leader_election.cc:302] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 989d2f289c3a4604a94c57d635989c94, dacea321aaf647318b088b32d465fd0a; no voters:
I20260926 15:04:19.653211 28247 raft_consensus.cc:2781] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:19.653399 28247 raft_consensus.cc:686] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94 [term 1 LEADER]: Becoming Leader. State: Replica: 989d2f289c3a4604a94c57d635989c94, State: Running, Role: LEADER
I20260926 15:04:19.653533 28247 consensus_queue.cc:237] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b19d303dec4492ea09fa238055aa4c4" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36797 } } peers { permanent_uuid: "989d2f289c3a4604a94c57d635989c94" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36059 } } peers { permanent_uuid: "dacea321aaf647318b088b32d465fd0a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41321 } }
I20260926 15:04:19.654436 27764 catalog_manager.cc:5495] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94 reported cstate change: term changed from 0 to 1, leader changed from <none> to 989d2f289c3a4604a94c57d635989c94 (127.8.102.1). New cstate: current_term: 1 leader_uuid: "989d2f289c3a4604a94c57d635989c94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b19d303dec4492ea09fa238055aa4c4" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36797 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "989d2f289c3a4604a94c57d635989c94" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 36059 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "dacea321aaf647318b088b32d465fd0a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41321 } health_report { overall_health: UNKNOWN } } }
W20260926 15:04:19.674991 28224 tablet.cc:2369] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:04:19.692040 28239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:19.689430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43854 (local address 127.8.102.1:36059)
0926 15:04:19.689506 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:19.689509 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:19.689519 (+ 10us) server_negotiation.cc:407] Connection header received
0926 15:04:19.689574 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:19.689575 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:19.689596 (+ 21us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:19.689684 (+ 88us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:04:19.689846 (+ 162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.690371 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.691203 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.691405 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.691497 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:19.691680 (+ 183us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:04:19.691790 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:04:19.691889 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:19.691909 (+ 20us) server_negotiation.cc:299] Negotiation successful
0926 15:04:19.691930 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 15:04:19.694119 28043 raft_consensus.cc:1261] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a [term 1 FOLLOWER]: Refusing update from remote peer 989d2f289c3a4604a94c57d635989c94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:04:19.694129 28177 raft_consensus.cc:1261] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4 [term 1 FOLLOWER]: Refusing update from remote peer 989d2f289c3a4604a94c57d635989c94: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:04:19.694545 28247 consensus_queue.cc:1035] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b19d303dec4492ea09fa238055aa4c4" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 36797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:04:19.694742 28251 consensus_queue.cc:1035] T f04241ad4f2a4cba809899ec5670b038 P 989d2f289c3a4604a94c57d635989c94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dacea321aaf647318b088b32d465fd0a" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41321 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:04:19.697383 28255 mvcc.cc:205] Tried to move back new op lower bound from 7333622004504121344 to 7333622004341477376. Current Snapshot: MvccSnapshot[applied={T|T < 7333622004504121344}]
I20260926 15:04:19.697399 28256 mvcc.cc:205] Tried to move back new op lower bound from 7333622004504121344 to 7333622004341477376. Current Snapshot: MvccSnapshot[applied={T|T < 7333622004504121344}]
I20260926 15:04:19.700416 27763 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:04:19.700544 27763 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:04:19.701402 27763 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f680e099a23f480287124e3ae5d7dd79: Sending DeleteTablet for 3 replicas of tablet f04241ad4f2a4cba809899ec5670b038
I20260926 15:04:19.701834 27889 tablet_service.cc:1514] Processing DeleteTablet for tablet f04241ad4f2a4cba809899ec5670b038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:04:19 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:43850
I20260926 15:04:19.701844 28157 tablet_service.cc:1514] Processing DeleteTablet for tablet f04241ad4f2a4cba809899ec5670b038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:04:19 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:41590
I20260926 15:04:19.702032 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 27826
I20260926 15:04:19.702128 28023 tablet_service.cc:1514] Processing DeleteTablet for tablet f04241ad4f2a4cba809899ec5670b038 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:04:19 UTC) from {username='kudu', principal='kudu/127.8.102.62@KRBTEST.COM'} at 127.0.0.1:47114
I20260926 15:04:19.702615 28266 tablet_replica.cc:330] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a: stopping tablet replica
I20260926 15:04:19.702746 28266 raft_consensus.cc:2229] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:04:19.702841 28265 tablet_replica.cc:330] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4: stopping tablet replica
I20260926 15:04:19.702867 28266 raft_consensus.cc:2258] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:04:19.702960 28265 raft_consensus.cc:2229] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:04:19.703071 28265 raft_consensus.cc:2258] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:04:19.704016 28266 ts_tablet_manager.cc:1905] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:04:19.705086 28265 ts_tablet_manager.cc:1905] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:04:19.706221 28266 ts_tablet_manager.cc:1918] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:04:19.706329 28266 log.cc:1196] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/f04241ad4f2a4cba809899ec5670b038
I20260926 15:04:19.706741 28266 ts_tablet_manager.cc:1939] T f04241ad4f2a4cba809899ec5670b038 P dacea321aaf647318b088b32d465fd0a: Deleting consensus metadata
I20260926 15:04:19.707280 28265 ts_tablet_manager.cc:1918] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:04:19.707373 28265 log.cc:1196] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/f04241ad4f2a4cba809899ec5670b038
I20260926 15:04:19.707348 27751 catalog_manager.cc:4841] TS dacea321aaf647318b088b32d465fd0a (127.8.102.2:41321): tablet f04241ad4f2a4cba809899ec5670b038 (table test-table [id=ec32af4db974434aa58107f818157943]) successfully deleted
I20260926 15:04:19.707669 28265 ts_tablet_manager.cc:1939] T f04241ad4f2a4cba809899ec5670b038 P 7b19d303dec4492ea09fa238055aa4c4: Deleting consensus metadata
I20260926 15:04:19.708135 27748 catalog_manager.cc:4841] TS 7b19d303dec4492ea09fa238055aa4c4 (127.8.102.3:36797): tablet f04241ad4f2a4cba809899ec5670b038 (table test-table [id=ec32af4db974434aa58107f818157943]) successfully deleted
W20260926 15:04:19.709120 27751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.102.1:36059 (error 108)
I20260926 15:04:19.709424 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 27960
I20260926 15:04:19.710368 28234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:19.709604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:36059 (local address 127.0.0.1:43862)
0926 15:04:19.710294 (+ 690us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:19.710337 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.1:36059: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":669}
W20260926 15:04:19.710543 27751 catalog_manager.cc:4584] TS 989d2f289c3a4604a94c57d635989c94 (127.8.102.1:36059): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f04241ad4f2a4cba809899ec5670b038: Network error: Client connection negotiation failed: client connection to 127.8.102.1:36059: connect: Connection refused (error 111)
I20260926 15:04:19.715902 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 28094
I20260926 15:04:19.720446 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 27732
2026-09-26T15:04:19Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (2200 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-26T15:04:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:19Z Disabled control of system clock
I20260926 15:04:19.748839 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:43371
--webserver_interface=127.8.102.62
--webserver_port=0
--builtin_ntp_servers=127.8.102.20:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:43371
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:04:19.835435 28271 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:19.835683 28271 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:19.835706 28271 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:19.836939 28271 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:19.836988 28271 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:19.837000 28271 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:19.837014 28271 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:19.837026 28271 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:19.838366 28271 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:43371
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:43371
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28271
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:19.838617 28271 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:19.838980 28271 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:19.844580 28276 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:19.844609 28279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:19.844573 28277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:19.844904 28271 server_base.cc:1034] running on GCE node
I20260926 15:04:19.845142 28271 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:19.845458 28271 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:19.846621 28271 hybrid_clock.cc:648] HybridClock initialized: now 1790435059846595 us; error 40 us; skew 500 ppm
I20260926 15:04:19.848014 28271 webserver.cc:460] Webserver started at http://127.8.102.62:36709/ using document root <none> and password file <none>
I20260926 15:04:19.848330 28271 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:19.848384 28271 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:19.848526 28271 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:19.849647 28271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "77c04a18d392475298f5d0f3c63f4531"
format_stamp: "Formatted at 2026-09-26 15:04:19 on dist-test-slave-jnhs"
server_key: "985e96597e866f70fb6642f696ceee52"
server_key_iv: "3d294052057d4fc3ee2b7ce3c7584bb6"
server_key_version: "encryptionkey@0"
I20260926 15:04:19.850044 28271 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "77c04a18d392475298f5d0f3c63f4531"
format_stamp: "Formatted at 2026-09-26 15:04:19 on dist-test-slave-jnhs"
server_key: "985e96597e866f70fb6642f696ceee52"
server_key_iv: "3d294052057d4fc3ee2b7ce3c7584bb6"
server_key_version: "encryptionkey@0"
I20260926 15:04:19.851532 28271 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 15:04:19.852420 28285 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:19.852690 28271 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:04:19.852773 28271 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "77c04a18d392475298f5d0f3c63f4531"
format_stamp: "Formatted at 2026-09-26 15:04:19 on dist-test-slave-jnhs"
server_key: "985e96597e866f70fb6642f696ceee52"
server_key_iv: "3d294052057d4fc3ee2b7ce3c7584bb6"
server_key_version: "encryptionkey@0"
I20260926 15:04:19.852880 28271 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:19.866837 28271 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:19.867192 28271 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:19.867372 28271 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:19.871796 28271 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:43371
I20260926 15:04:19.871829 28337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:43371 every 8 connection(s)
I20260926 15:04:19.872198 28271 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:19.873365 28338 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:19.873566 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 28271
I20260926 15:04:19.873693 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:19.873911 8600 external_mini_cluster.cc:1428] Setting key b274bc7354ac455ad14c68dcbce4c478
I20260926 15:04:19.877094 28338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531: Bootstrap starting.
I20260926 15:04:19.877787 28338 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:19.878113 28338 log.cc:824] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:19.878942 28338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531: No bootstrap required, opened a new log
I20260926 15:04:19.880010 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:19.874501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34312 (local address 127.8.102.62:43371)
0926 15:04:19.874723 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:19.874731 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:19.874766 (+ 35us) server_negotiation.cc:407] Connection header received
0926 15:04:19.875229 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:19.875236 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:19.875299 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:19.875414 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:19.875687 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.876601 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.877713 (+ 1112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:19.878292 (+ 579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:19.878457 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:19.879248 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:19.879252 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:19.879253 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:19.879457 (+ 204us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:19.879503 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:19.879519 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:19.879604 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:19.879696 (+ 92us) server_negotiation.cc:299] Negotiation successful
0926 15:04:19.879785 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260926 15:04:19.881327 28338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c04a18d392475298f5d0f3c63f4531" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43371 } }
I20260926 15:04:19.881469 28338 raft_consensus.cc:374] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:19.881491 28338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 77c04a18d392475298f5d0f3c63f4531, State: Initialized, Role: FOLLOWER
I20260926 15:04:19.881620 28338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c04a18d392475298f5d0f3c63f4531" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43371 } }
I20260926 15:04:19.881704 28338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:19.881727 28338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:19.881778 28338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:19.882370 28338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c04a18d392475298f5d0f3c63f4531" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43371 } }
I20260926 15:04:19.882455 28338 leader_election.cc:302] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 77c04a18d392475298f5d0f3c63f4531; no voters:
I20260926 15:04:19.882714 28338 leader_election.cc:288] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:19.882860 28343 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:19.883071 28343 raft_consensus.cc:686] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531 [term 1 LEADER]: Becoming Leader. State: Replica: 77c04a18d392475298f5d0f3c63f4531, State: Running, Role: LEADER
I20260926 15:04:19.883118 28338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:19.883292 28343 consensus_queue.cc:237] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c04a18d392475298f5d0f3c63f4531" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43371 } }
I20260926 15:04:19.883702 28344 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "77c04a18d392475298f5d0f3c63f4531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c04a18d392475298f5d0f3c63f4531" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43371 } } }
I20260926 15:04:19.883706 28345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 77c04a18d392475298f5d0f3c63f4531. Latest consensus state: current_term: 1 leader_uuid: "77c04a18d392475298f5d0f3c63f4531" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "77c04a18d392475298f5d0f3c63f4531" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 43371 } } }
I20260926 15:04:19.883822 28344 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:19.883829 28345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:19.884325 28350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:19.884848 28350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:19.886525 28350 catalog_manager.cc:1347] Generated new cluster ID: ded12874340f49a680d5e4b8a1ff9900
I20260926 15:04:19.886584 28350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:19.895036 28350 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:19.895634 28350 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:19.898247 28350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531: Generated new TSK 0
I20260926 15:04:19.898469 28350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:19.900408 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.1:0
--local_ip_for_outbound_sockets=127.8.102.1
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:43371
--builtin_ntp_servers=127.8.102.20:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:04:19.983541 28362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:19.983872 28362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:19.983914 28362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:19.985370 28362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:19.985435 28362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:19.985477 28362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.1
I20260926 15:04:19.986907 28362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.8.102.1
--webserver_port=0
--tserver_master_addrs=127.8.102.62:43371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.1
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:19.987183 28362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:19.987501 28362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:19.994199 28368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:19.994256 28362 server_base.cc:1034] running on GCE node
W20260926 15:04:19.994172 28370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:19.994401 28367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:19.994745 28362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:19.995038 28362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:19.996272 28362 hybrid_clock.cc:648] HybridClock initialized: now 1790435059996248 us; error 45 us; skew 500 ppm
I20260926 15:04:19.997761 28362 webserver.cc:460] Webserver started at http://127.8.102.1:35513/ using document root <none> and password file <none>
I20260926 15:04:19.998054 28362 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:19.998102 28362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:19.998248 28362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:19.999513 28362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-0/data/instance:
uuid: "ca151c9f3fe948f9a114c2f99246831a"
format_stamp: "Formatted at 2026-09-26 15:04:19 on dist-test-slave-jnhs"
server_key: "d71a8ef920c9ab99bcebb85f9bf00837"
server_key_iv: "a42f88ef3f42d39891fed736a7a31fbb"
server_key_version: "encryptionkey@0"
I20260926 15:04:19.999895 28362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance:
uuid: "ca151c9f3fe948f9a114c2f99246831a"
format_stamp: "Formatted at 2026-09-26 15:04:19 on dist-test-slave-jnhs"
server_key: "d71a8ef920c9ab99bcebb85f9bf00837"
server_key_iv: "a42f88ef3f42d39891fed736a7a31fbb"
server_key_version: "encryptionkey@0"
I20260926 15:04:20.001320 28362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 15:04:20.002197 28376 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:20.002480 28362 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:20.002600 28362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-0/wal
uuid: "ca151c9f3fe948f9a114c2f99246831a"
format_stamp: "Formatted at 2026-09-26 15:04:19 on dist-test-slave-jnhs"
server_key: "d71a8ef920c9ab99bcebb85f9bf00837"
server_key_iv: "a42f88ef3f42d39891fed736a7a31fbb"
server_key_version: "encryptionkey@0"
I20260926 15:04:20.002697 28362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:20.020782 28362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:20.021164 28362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:20.021337 28362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:20.021600 28362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:04:20.021971 28362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:04:20.022032 28362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:20.022105 28362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:04:20.022143 28362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:20.028273 28362 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.1:32983
I20260926 15:04:20.028362 28489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.1:32983 every 8 connection(s)
I20260926 15:04:20.028718 28362 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-0/data/info.pb
I20260926 15:04:20.034736 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:20.029751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.1:58697 (local address 127.8.102.62:43371)
0926 15:04:20.029822 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:20.029824 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:20.030079 (+ 255us) server_negotiation.cc:407] Connection header received
0926 15:04:20.030581 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:20.030582 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:20.030596 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:20.030741 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:20.031514 (+ 773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.032060 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.033082 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.033319 (+ 237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.033385 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:20.034316 (+ 931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:20.034318 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:20.034318 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:20.034323 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:20.034339 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:20.034343 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:20.034413 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:20.034553 (+ 140us) server_negotiation.cc:299] Negotiation successful
0926 15:04:20.034582 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 15:04:20.034862 28492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:20.029710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:43371 (local address 127.8.102.1:58697)
0926 15:04:20.029953 (+ 243us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:20.029989 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:20.030473 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:20.030823 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:20.030827 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:20.030835 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:20.031391 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:20.031396 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.032152 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.032153 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:20.032990 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:20.032992 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.033401 (+ 409us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.033402 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:20.033515 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:20.034156 (+ 641us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:04:20.034156 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:20.034186 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:04:20.034212 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:20.034420 (+ 208us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:20.034425 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:20.034427 (+ 2us) client_negotiation.cc:769] Sending connection context
0926 15:04:20.034481 (+ 54us) client_negotiation.cc:240] Negotiation successful
0926 15:04:20.034568 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":112,"threads_started":1}
I20260926 15:04:20.035430 28490 heartbeater.cc:347] Connected to a master server at 127.8.102.62:43371
I20260926 15:04:20.035544 28490 heartbeater.cc:464] Registering TS with master...
I20260926 15:04:20.035751 28490 heartbeater.cc:511] Master 127.8.102.62:43371 requested a full tablet report, sending...
I20260926 15:04:20.036171 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 28362
I20260926 15:04:20.036279 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-0/wal/instance
I20260926 15:04:20.036401 28302 ts_manager.cc:194] Registered new tserver with Master: ca151c9f3fe948f9a114c2f99246831a (127.8.102.1:32983)
I20260926 15:04:20.036530 8600 external_mini_cluster.cc:1428] Setting key fd30a4d30ae381b396c19275b1da221d
I20260926 15:04:20.037254 28302 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.102.1:58697
I20260926 15:04:20.037791 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.2:0
--local_ip_for_outbound_sockets=127.8.102.2
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:43371
--builtin_ntp_servers=127.8.102.20:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260926 15:04:20.119488 28493 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:20.119699 28493 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:20.119721 28493 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:20.120954 28493 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:20.121002 28493 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:20.121029 28493 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.2
I20260926 15:04:20.122413 28493 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.8.102.2
--webserver_port=0
--tserver_master_addrs=127.8.102.62:43371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28493
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.2
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:20.122692 28493 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:20.123030 28493 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:20.128132 28501 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:20.128305 28499 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:20.128358 28493 server_base.cc:1034] running on GCE node
W20260926 15:04:20.128305 28498 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:20.128697 28493 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:20.128975 28493 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:20.130187 28493 hybrid_clock.cc:648] HybridClock initialized: now 1790435060130163 us; error 34 us; skew 500 ppm
I20260926 15:04:20.131639 28493 webserver.cc:460] Webserver started at http://127.8.102.2:39123/ using document root <none> and password file <none>
I20260926 15:04:20.131906 28493 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:20.131976 28493 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:20.132164 28493 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:20.133296 28493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-1/data/instance:
uuid: "4c1f13c083644b05bb8d258385362537"
format_stamp: "Formatted at 2026-09-26 15:04:20 on dist-test-slave-jnhs"
server_key: "82486baff2251110431c0eea7ebed5bd"
server_key_iv: "e1c1bf985fa3f4cd07394a044cf824bd"
server_key_version: "encryptionkey@0"
I20260926 15:04:20.133675 28493 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance:
uuid: "4c1f13c083644b05bb8d258385362537"
format_stamp: "Formatted at 2026-09-26 15:04:20 on dist-test-slave-jnhs"
server_key: "82486baff2251110431c0eea7ebed5bd"
server_key_iv: "e1c1bf985fa3f4cd07394a044cf824bd"
server_key_version: "encryptionkey@0"
I20260926 15:04:20.135299 28493 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260926 15:04:20.136077 28507 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:20.136276 28493 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260926 15:04:20.136377 28493 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-1/wal
uuid: "4c1f13c083644b05bb8d258385362537"
format_stamp: "Formatted at 2026-09-26 15:04:20 on dist-test-slave-jnhs"
server_key: "82486baff2251110431c0eea7ebed5bd"
server_key_iv: "e1c1bf985fa3f4cd07394a044cf824bd"
server_key_version: "encryptionkey@0"
I20260926 15:04:20.136461 28493 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:20.149855 28493 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:20.150199 28493 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:20.150354 28493 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:20.150594 28493 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:04:20.150966 28493 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:04:20.151021 28493 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:20.151062 28493 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:04:20.151091 28493 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:20.157763 28493 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.2:41151
I20260926 15:04:20.157853 28620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.2:41151 every 8 connection(s)
I20260926 15:04:20.158202 28493 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-1/data/info.pb
I20260926 15:04:20.163210 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 28493
I20260926 15:04:20.163249 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:20.158982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:41597 (local address 127.8.102.62:43371)
0926 15:04:20.159078 (+ 96us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:20.159080 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:20.159306 (+ 226us) server_negotiation.cc:407] Connection header received
0926 15:04:20.159757 (+ 451us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:20.159758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:20.159768 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:20.159828 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:20.160473 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.160950 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.161972 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.162152 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.162210 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:20.162988 (+ 778us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:20.162989 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:20.162989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:20.162994 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:20.163007 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:20.163010 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:20.163053 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:20.163158 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 15:04:20.163181 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260926 15:04:20.163331 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-1/wal/instance
I20260926 15:04:20.163579 8600 external_mini_cluster.cc:1428] Setting key a8624185d80f3b3a693624c05494ff97
I20260926 15:04:20.163583 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:20.158977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:43371 (local address 127.8.102.2:41597)
0926 15:04:20.159150 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:20.159192 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:20.159673 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:20.159885 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:20.159888 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:20.159896 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:20.160401 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:20.160405 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.161082 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.161083 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:20.161892 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:20.161894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.162226 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.162227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:20.162334 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:20.162850 (+ 516us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:04:20.162850 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:20.162879 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:04:20.162903 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:20.163070 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:20.163075 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:20.163078 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:04:20.163155 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 15:04:20.163279 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
I20260926 15:04:20.164179 28621 heartbeater.cc:347] Connected to a master server at 127.8.102.62:43371
I20260926 15:04:20.164326 28621 heartbeater.cc:464] Registering TS with master...
I20260926 15:04:20.164515 28621 heartbeater.cc:511] Master 127.8.102.62:43371 requested a full tablet report, sending...
I20260926 15:04:20.164564 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.8.102.3:0
--local_ip_for_outbound_sockets=127.8.102.3
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:43371
--builtin_ntp_servers=127.8.102.20:37085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260926 15:04:20.165001 28302 ts_manager.cc:194] Registered new tserver with Master: 4c1f13c083644b05bb8d258385362537 (127.8.102.2:41151)
I20260926 15:04:20.165506 28302 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.102.2:41597
W20260926 15:04:20.249737 28624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:20.249946 28624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:20.249969 28624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:20.251236 28624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:20.251282 28624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:20.251307 28624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.8.102.3
I20260926 15:04:20.252697 28624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:37085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.8.102.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.8.102.3
--webserver_port=0
--tserver_master_addrs=127.8.102.62:43371
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28624
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.8.102.3
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:20.252957 28624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:20.253301 28624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:20.258666 28629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:20.258967 28630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:20.258992 28632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:20.259076 28624 server_base.cc:1034] running on GCE node
I20260926 15:04:20.259298 28624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:20.259588 28624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:20.260768 28624 hybrid_clock.cc:648] HybridClock initialized: now 1790435060260755 us; error 29 us; skew 500 ppm
I20260926 15:04:20.262081 28624 webserver.cc:460] Webserver started at http://127.8.102.3:36937/ using document root <none> and password file <none>
I20260926 15:04:20.262362 28624 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:20.262434 28624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:20.262588 28624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:20.263751 28624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-2/data/instance:
uuid: "06e6bc4675354394a152c911f53de189"
format_stamp: "Formatted at 2026-09-26 15:04:20 on dist-test-slave-jnhs"
server_key: "987f8985e00b0f63f02cc825309e48d4"
server_key_iv: "eb038d4b9ca54f1cb0036c38ea9d9537"
server_key_version: "encryptionkey@0"
I20260926 15:04:20.264135 28624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance:
uuid: "06e6bc4675354394a152c911f53de189"
format_stamp: "Formatted at 2026-09-26 15:04:20 on dist-test-slave-jnhs"
server_key: "987f8985e00b0f63f02cc825309e48d4"
server_key_iv: "eb038d4b9ca54f1cb0036c38ea9d9537"
server_key_version: "encryptionkey@0"
I20260926 15:04:20.265583 28624 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 15:04:20.266435 28638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:20.266710 28624 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:04:20.266814 28624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-2/wal
uuid: "06e6bc4675354394a152c911f53de189"
format_stamp: "Formatted at 2026-09-26 15:04:20 on dist-test-slave-jnhs"
server_key: "987f8985e00b0f63f02cc825309e48d4"
server_key_iv: "eb038d4b9ca54f1cb0036c38ea9d9537"
server_key_version: "encryptionkey@0"
I20260926 15:04:20.266898 28624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:20.280802 28624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:20.281141 28624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:20.281292 28624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:20.281519 28624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260926 15:04:20.281862 28624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260926 15:04:20.281919 28624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:20.281965 28624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260926 15:04:20.282001 28624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:20.288534 28624 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.3:38319
I20260926 15:04:20.288618 28751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.3:38319 every 8 connection(s)
I20260926 15:04:20.288987 28624 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-2/data/info.pb
I20260926 15:04:20.290011 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 28624
I20260926 15:04:20.290148 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-2/wal/instance
I20260926 15:04:20.290380 8600 external_mini_cluster.cc:1428] Setting key b255a3afca212549da06e20f1ab462fe
I20260926 15:04:20.295097 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:20.289884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.3:45161 (local address 127.8.102.62:43371)
0926 15:04:20.289949 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:20.289951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:20.290279 (+ 328us) server_negotiation.cc:407] Connection header received
0926 15:04:20.290944 (+ 665us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:20.290945 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:20.290957 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:20.291029 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:20.291890 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.292400 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.293520 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.293721 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.293781 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:20.294806 (+ 1025us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:20.294808 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:20.294808 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:20.294813 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:20.294828 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:20.294831 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:20.294879 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:20.294969 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 15:04:20.294993 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 15:04:20.295279 28754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:20.289819 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:43371 (local address 127.8.102.3:45161)
0926 15:04:20.290124 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:20.290170 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:20.290847 (+ 677us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:20.291160 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:20.291164 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:20.291175 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:20.291797 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:20.291803 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.292532 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.292533 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:20.293422 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:20.293425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.293824 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.293825 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:20.293944 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:20.294563 (+ 619us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0926 15:04:20.294563 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:20.294594 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0926 15:04:20.294618 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:20.294895 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:20.294900 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:20.294903 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:04:20.294959 (+ 56us) client_negotiation.cc:240] Negotiation successful
0926 15:04:20.295047 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":231,"thread_start_us":92,"threads_started":1}
I20260926 15:04:20.295790 28752 heartbeater.cc:347] Connected to a master server at 127.8.102.62:43371
I20260926 15:04:20.295872 28752 heartbeater.cc:464] Registering TS with master...
I20260926 15:04:20.296026 28752 heartbeater.cc:511] Master 127.8.102.62:43371 requested a full tablet report, sending...
I20260926 15:04:20.296350 28302 ts_manager.cc:194] Registered new tserver with Master: 06e6bc4675354394a152c911f53de189 (127.8.102.3:38319)
I20260926 15:04:20.296731 28302 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.8.102.3:45161
I20260926 15:04:20.302569 8600 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260926 15:04:20.307446 28341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:20.304140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34324 (local address 127.8.102.62:43371)
0926 15:04:20.304216 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:20.304218 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:20.304323 (+ 105us) server_negotiation.cc:407] Connection header received
0926 15:04:20.304414 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:20.304415 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:20.304426 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:20.304484 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:20.304680 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.305305 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.306102 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.306359 (+ 257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.306444 (+ 85us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:20.307204 (+ 760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:20.307206 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:20.307207 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:20.307212 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:20.307228 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:20.307232 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:20.307281 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:20.307329 (+ 48us) server_negotiation.cc:299] Negotiation successful
0926 15:04:20.307354 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260926 15:04:20.309232 28302 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260926 15:04:20.310822 28302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260926 15:04:20.317412 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:20.313227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42088 (local address 127.8.102.2:41151)
0926 15:04:20.313440 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:20.313447 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:20.313668 (+ 221us) server_negotiation.cc:407] Connection header received
0926 15:04:20.313712 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:20.313714 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:20.313727 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:20.313826 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:20.314009 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.314925 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.316342 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.316982 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.317088 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:20.317187 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:20.317277 (+ 90us) server_negotiation.cc:299] Negotiation successful
0926 15:04:20.317315 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":129,"threads_started":1}
I20260926 15:04:20.317399 28760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:20.313085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:41151 (local address 127.0.0.1:42088)
0926 15:04:20.313551 (+ 466us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:20.313561 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:20.313609 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:20.313833 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:20.313836 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:20.313838 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:20.313921 (+ 83us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:20.313923 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.315011 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.315012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:20.316253 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:20.316255 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.317091 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.317092 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:20.317172 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:20.317185 (+ 13us) client_negotiation.cc:769] Sending connection context
0926 15:04:20.317262 (+ 77us) client_negotiation.cc:240] Negotiation successful
0926 15:04:20.317315 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"mutex_wait_us":5,"thread_start_us":110,"threads_started":1}
I20260926 15:04:20.317865 28765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:20.313519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56932 (local address 127.8.102.3:38319)
0926 15:04:20.313691 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:20.313696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:20.314043 (+ 347us) server_negotiation.cc:407] Connection header received
0926 15:04:20.314086 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:20.314088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:20.314102 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:20.314167 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:20.314365 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.315460 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.316669 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.317376 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.317439 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:20.317537 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:20.317732 (+ 195us) server_negotiation.cc:299] Negotiation successful
0926 15:04:20.317785 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":95,"threads_started":1}
I20260926 15:04:20.317962 28761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:20.313143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:38319 (local address 127.0.0.1:56932)
0926 15:04:20.313918 (+ 775us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:20.313928 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:20.313973 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:20.314231 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:20.314232 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:20.314234 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:20.314276 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:20.314277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.315553 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.315554 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:20.316580 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:20.316583 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.317573 (+ 990us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.317575 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:20.317638 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:20.317641 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:04:20.317698 (+ 57us) client_negotiation.cc:240] Negotiation successful
0926 15:04:20.317858 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":108,"threads_started":1}
I20260926 15:04:20.318277 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:20.313382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52466 (local address 127.8.102.1:32983)
0926 15:04:20.313740 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:20.313743 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:20.313753 (+ 10us) server_negotiation.cc:407] Connection header received
0926 15:04:20.313800 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:20.313801 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:20.313813 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:20.313910 (+ 97us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:20.314119 (+ 209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.315281 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.316511 (+ 1230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.317240 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.317315 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:20.317399 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:20.318142 (+ 743us) server_negotiation.cc:299] Negotiation successful
0926 15:04:20.318188 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":323,"thread_start_us":84,"threads_started":1}
I20260926 15:04:20.318286 28555 tablet_service.cc:1467] Processing CreateTablet for tablet 3b2a7ad845454fae9c8e1330f2546141 (DEFAULT_TABLE table=test-table [id=38cba85918f54f74b2aea76326b22803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:04:20.318508 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:20.313085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:32983 (local address 127.0.0.1:52466)
0926 15:04:20.313354 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:20.313401 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:20.313479 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:20.313987 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:20.313988 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:20.313990 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:20.314038 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:20.314040 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.315370 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.315372 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:20.316424 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:20.316426 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.317437 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.317438 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:20.317859 (+ 421us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:20.317862 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:04:20.318382 (+ 520us) client_negotiation.cc:240] Negotiation successful
0926 15:04:20.318430 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":233,"mutex_wait_us":29,"thread_start_us":107,"threads_started":1}
I20260926 15:04:20.318718 28555 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b2a7ad845454fae9c8e1330f2546141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:20.318701 28686 tablet_service.cc:1467] Processing CreateTablet for tablet 3b2a7ad845454fae9c8e1330f2546141 (DEFAULT_TABLE table=test-table [id=38cba85918f54f74b2aea76326b22803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:04:20.319063 28686 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b2a7ad845454fae9c8e1330f2546141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:20.319180 28424 tablet_service.cc:1467] Processing CreateTablet for tablet 3b2a7ad845454fae9c8e1330f2546141 (DEFAULT_TABLE table=test-table [id=38cba85918f54f74b2aea76326b22803]), partition=RANGE (key) PARTITION UNBOUNDED
I20260926 15:04:20.319527 28424 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3b2a7ad845454fae9c8e1330f2546141. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:20.322018 28766 tablet_bootstrap.cc:492] T 3b2a7ad845454fae9c8e1330f2546141 P ca151c9f3fe948f9a114c2f99246831a: Bootstrap starting.
I20260926 15:04:20.322628 28768 tablet_bootstrap.cc:492] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537: Bootstrap starting.
I20260926 15:04:20.322777 28767 tablet_bootstrap.cc:492] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189: Bootstrap starting.
I20260926 15:04:20.322860 28766 tablet_bootstrap.cc:654] T 3b2a7ad845454fae9c8e1330f2546141 P ca151c9f3fe948f9a114c2f99246831a: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:20.323155 28766 log.cc:824] T 3b2a7ad845454fae9c8e1330f2546141 P ca151c9f3fe948f9a114c2f99246831a: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:20.323354 28768 tablet_bootstrap.cc:654] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:20.323354 28767 tablet_bootstrap.cc:654] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:20.323689 28767 log.cc:824] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:20.323689 28768 log.cc:824] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:20.324005 28766 tablet_bootstrap.cc:492] T 3b2a7ad845454fae9c8e1330f2546141 P ca151c9f3fe948f9a114c2f99246831a: No bootstrap required, opened a new log
I20260926 15:04:20.324088 28766 ts_tablet_manager.cc:1397] T 3b2a7ad845454fae9c8e1330f2546141 P ca151c9f3fe948f9a114c2f99246831a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260926 15:04:20.324455 28767 tablet_bootstrap.cc:492] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189: No bootstrap required, opened a new log
I20260926 15:04:20.324577 28767 ts_tablet_manager.cc:1397] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:04:20.324728 28768 tablet_bootstrap.cc:492] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537: No bootstrap required, opened a new log
I20260926 15:04:20.324793 28768 ts_tablet_manager.cc:1397] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260926 15:04:20.326359 28766 raft_consensus.cc:348] T 3b2a7ad845454fae9c8e1330f2546141 P ca151c9f3fe948f9a114c2f99246831a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1f13c083644b05bb8d258385362537" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41151 } } peers { permanent_uuid: "06e6bc4675354394a152c911f53de189" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 38319 } } peers { permanent_uuid: "ca151c9f3fe948f9a114c2f99246831a" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 32983 } }
I20260926 15:04:20.326505 28766 raft_consensus.cc:374] T 3b2a7ad845454fae9c8e1330f2546141 P ca151c9f3fe948f9a114c2f99246831a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:20.326529 28766 raft_consensus.cc:729] T 3b2a7ad845454fae9c8e1330f2546141 P ca151c9f3fe948f9a114c2f99246831a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ca151c9f3fe948f9a114c2f99246831a, State: Initialized, Role: FOLLOWER
I20260926 15:04:20.326728 28766 consensus_queue.cc:260] T 3b2a7ad845454fae9c8e1330f2546141 P ca151c9f3fe948f9a114c2f99246831a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1f13c083644b05bb8d258385362537" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41151 } } peers { permanent_uuid: "06e6bc4675354394a152c911f53de189" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 38319 } } peers { permanent_uuid: "ca151c9f3fe948f9a114c2f99246831a" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 32983 } }
I20260926 15:04:20.327018 28766 ts_tablet_manager.cc:1428] T 3b2a7ad845454fae9c8e1330f2546141 P ca151c9f3fe948f9a114c2f99246831a: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260926 15:04:20.327163 28490 heartbeater.cc:503] Master 127.8.102.62:43371 was elected leader, sending a full tablet report...
I20260926 15:04:20.327466 28768 raft_consensus.cc:348] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1f13c083644b05bb8d258385362537" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41151 } } peers { permanent_uuid: "06e6bc4675354394a152c911f53de189" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 38319 } } peers { permanent_uuid: "ca151c9f3fe948f9a114c2f99246831a" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 32983 } }
I20260926 15:04:20.327636 28768 raft_consensus.cc:374] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:20.327685 28768 raft_consensus.cc:729] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c1f13c083644b05bb8d258385362537, State: Initialized, Role: FOLLOWER
I20260926 15:04:20.327535 28767 raft_consensus.cc:348] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1f13c083644b05bb8d258385362537" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41151 } } peers { permanent_uuid: "06e6bc4675354394a152c911f53de189" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 38319 } } peers { permanent_uuid: "ca151c9f3fe948f9a114c2f99246831a" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 32983 } }
I20260926 15:04:20.327759 28767 raft_consensus.cc:374] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:20.327824 28767 raft_consensus.cc:729] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 06e6bc4675354394a152c911f53de189, State: Initialized, Role: FOLLOWER
I20260926 15:04:20.327811 28768 consensus_queue.cc:260] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1f13c083644b05bb8d258385362537" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41151 } } peers { permanent_uuid: "06e6bc4675354394a152c911f53de189" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 38319 } } peers { permanent_uuid: "ca151c9f3fe948f9a114c2f99246831a" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 32983 } }
I20260926 15:04:20.327960 28767 consensus_queue.cc:260] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1f13c083644b05bb8d258385362537" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41151 } } peers { permanent_uuid: "06e6bc4675354394a152c911f53de189" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 38319 } } peers { permanent_uuid: "ca151c9f3fe948f9a114c2f99246831a" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 32983 } }
I20260926 15:04:20.328075 28768 ts_tablet_manager.cc:1428] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260926 15:04:20.328197 28621 heartbeater.cc:503] Master 127.8.102.62:43371 was elected leader, sending a full tablet report...
I20260926 15:04:20.328207 28767 ts_tablet_manager.cc:1428] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260926 15:04:20.328311 28752 heartbeater.cc:503] Master 127.8.102.62:43371 was elected leader, sending a full tablet report...
W20260926 15:04:20.408821 28622 tablet.cc:2369] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:04:20.524058 28773 raft_consensus.cc:482] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260926 15:04:20.524196 28773 raft_consensus.cc:504] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1f13c083644b05bb8d258385362537" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41151 } } peers { permanent_uuid: "06e6bc4675354394a152c911f53de189" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 38319 } } peers { permanent_uuid: "ca151c9f3fe948f9a114c2f99246831a" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 32983 } }
I20260926 15:04:20.524574 28773 leader_election.cc:288] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 06e6bc4675354394a152c911f53de189 (127.8.102.3:38319), ca151c9f3fe948f9a114c2f99246831a (127.8.102.1:32983)
I20260926 15:04:20.528896 28764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:20.524874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:45357 (local address 127.8.102.1:32983)
0926 15:04:20.524927 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:20.524929 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:20.525129 (+ 200us) server_negotiation.cc:407] Connection header received
0926 15:04:20.525168 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:20.525169 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:20.525180 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:20.525226 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:20.525386 (+ 160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.526185 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.527824 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.528438 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.528547 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:20.528562 (+ 15us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:20.528767 (+ 205us) server_negotiation.cc:299] Negotiation successful
0926 15:04:20.528814 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260926 15:04:20.528896 28765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:20.524900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.8.102.2:59739 (local address 127.8.102.3:38319)
0926 15:04:20.524948 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:20.524950 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:20.524959 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:04:20.525001 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:20.525002 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:20.525019 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:20.525076 (+ 57us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0926 15:04:20.525331 (+ 255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.526201 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.527824 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.528511 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.528583 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:20.528592 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:20.528766 (+ 174us) server_negotiation.cc:299] Negotiation successful
0926 15:04:20.528815 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260926 15:04:20.528898 28623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:20.524793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.3:38319 (local address 127.8.102.2:59739)
0926 15:04:20.524847 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:20.524859 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:20.524911 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:20.525157 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:20.525158 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:20.525160 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:20.525251 (+ 91us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:20.525252 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.526301 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.526302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:20.527706 (+ 1404us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:20.527710 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.528595 (+ 885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.528596 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:20.528666 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:20.528672 (+ 6us) client_negotiation.cc:769] Sending connection context
0926 15:04:20.528735 (+ 63us) client_negotiation.cc:240] Negotiation successful
0926 15:04:20.528789 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32,"mutex_wait_us":45}
I20260926 15:04:20.528904 28775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:20.524804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:32983 (local address 127.8.102.2:45357)
0926 15:04:20.525018 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:20.525027 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:20.525105 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:20.525246 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:20.525247 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:20.525248 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0926 15:04:20.525310 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:20.525311 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.526300 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.526302 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:20.527708 (+ 1406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:20.527710 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.528574 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.528575 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:20.528673 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:20.528676 (+ 3us) client_negotiation.cc:769] Sending connection context
0926 15:04:20.528735 (+ 59us) client_negotiation.cc:240] Negotiation successful
0926 15:04:20.528787 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198,"mutex_wait_us":168,"thread_start_us":105,"threads_started":1}
I20260926 15:04:20.529356 28706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b2a7ad845454fae9c8e1330f2546141" candidate_uuid: "4c1f13c083644b05bb8d258385362537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e6bc4675354394a152c911f53de189" is_pre_election: true
W20260926 15:04:20.529475 28491 tablet.cc:2369] T 3b2a7ad845454fae9c8e1330f2546141 P ca151c9f3fe948f9a114c2f99246831a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:04:20.529537 28706 raft_consensus.cc:2454] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1f13c083644b05bb8d258385362537 in term 0.
I20260926 15:04:20.529693 28444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b2a7ad845454fae9c8e1330f2546141" candidate_uuid: "4c1f13c083644b05bb8d258385362537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca151c9f3fe948f9a114c2f99246831a" is_pre_election: true
I20260926 15:04:20.529781 28508 leader_election.cc:302] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 06e6bc4675354394a152c911f53de189, 4c1f13c083644b05bb8d258385362537; no voters:
I20260926 15:04:20.529870 28444 raft_consensus.cc:2454] T 3b2a7ad845454fae9c8e1330f2546141 P ca151c9f3fe948f9a114c2f99246831a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4c1f13c083644b05bb8d258385362537 in term 0.
I20260926 15:04:20.529933 28773 raft_consensus.cc:2781] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260926 15:04:20.529997 28773 raft_consensus.cc:482] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260926 15:04:20.530030 28773 raft_consensus.cc:3037] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:20.530923 28773 raft_consensus.cc:504] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1f13c083644b05bb8d258385362537" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41151 } } peers { permanent_uuid: "06e6bc4675354394a152c911f53de189" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 38319 } } peers { permanent_uuid: "ca151c9f3fe948f9a114c2f99246831a" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 32983 } }
I20260926 15:04:20.531098 28773 leader_election.cc:288] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [CANDIDATE]: Term 1 election: Requested vote from peers 06e6bc4675354394a152c911f53de189 (127.8.102.3:38319), ca151c9f3fe948f9a114c2f99246831a (127.8.102.1:32983)
I20260926 15:04:20.531260 28706 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b2a7ad845454fae9c8e1330f2546141" candidate_uuid: "4c1f13c083644b05bb8d258385362537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "06e6bc4675354394a152c911f53de189"
I20260926 15:04:20.531347 28444 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3b2a7ad845454fae9c8e1330f2546141" candidate_uuid: "4c1f13c083644b05bb8d258385362537" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ca151c9f3fe948f9a114c2f99246831a"
I20260926 15:04:20.531368 28706 raft_consensus.cc:3037] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:20.531457 28444 raft_consensus.cc:3037] T 3b2a7ad845454fae9c8e1330f2546141 P ca151c9f3fe948f9a114c2f99246831a [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:20.532260 28444 raft_consensus.cc:2454] T 3b2a7ad845454fae9c8e1330f2546141 P ca151c9f3fe948f9a114c2f99246831a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1f13c083644b05bb8d258385362537 in term 1.
I20260926 15:04:20.532275 28706 raft_consensus.cc:2454] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4c1f13c083644b05bb8d258385362537 in term 1.
I20260926 15:04:20.532441 28509 leader_election.cc:302] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4c1f13c083644b05bb8d258385362537, ca151c9f3fe948f9a114c2f99246831a; no voters:
I20260926 15:04:20.532558 28773 raft_consensus.cc:2781] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:20.532717 28773 raft_consensus.cc:686] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [term 1 LEADER]: Becoming Leader. State: Replica: 4c1f13c083644b05bb8d258385362537, State: Running, Role: LEADER
I20260926 15:04:20.532823 28773 consensus_queue.cc:237] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1f13c083644b05bb8d258385362537" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41151 } } peers { permanent_uuid: "06e6bc4675354394a152c911f53de189" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 38319 } } peers { permanent_uuid: "ca151c9f3fe948f9a114c2f99246831a" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 32983 } }
I20260926 15:04:20.533725 28301 catalog_manager.cc:5495] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4c1f13c083644b05bb8d258385362537 (127.8.102.2). New cstate: current_term: 1 leader_uuid: "4c1f13c083644b05bb8d258385362537" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4c1f13c083644b05bb8d258385362537" member_type: VOTER last_known_addr { host: "127.8.102.2" port: 41151 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "06e6bc4675354394a152c911f53de189" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 38319 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ca151c9f3fe948f9a114c2f99246831a" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 32983 } health_report { overall_health: UNKNOWN } } }
W20260926 15:04:20.539563 28753 tablet.cc:2369] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:04:20.548445 28763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:20.545954 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42104 (local address 127.8.102.2:41151)
0926 15:04:20.545996 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:20.545998 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:20.546029 (+ 31us) server_negotiation.cc:407] Connection header received
0926 15:04:20.546132 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:20.546132 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:20.546149 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:20.546212 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:04:20.546340 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.546893 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.547717 (+ 824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:20.547902 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:20.547962 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:20.548094 (+ 132us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:04:20.548194 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:04:20.548258 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:20.548342 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 15:04:20.548357 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 15:04:20.550230 28444 raft_consensus.cc:1261] T 3b2a7ad845454fae9c8e1330f2546141 P ca151c9f3fe948f9a114c2f99246831a [term 1 FOLLOWER]: Refusing update from remote peer 4c1f13c083644b05bb8d258385362537: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:04:20.550230 28706 raft_consensus.cc:1261] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189 [term 1 FOLLOWER]: Refusing update from remote peer 4c1f13c083644b05bb8d258385362537: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:04:20.550598 28773 consensus_queue.cc:1035] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [LEADER]: Connected to new peer: Peer: permanent_uuid: "06e6bc4675354394a152c911f53de189" member_type: VOTER last_known_addr { host: "127.8.102.3" port: 38319 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:04:20.550817 28776 consensus_queue.cc:1035] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ca151c9f3fe948f9a114c2f99246831a" member_type: VOTER last_known_addr { host: "127.8.102.1" port: 32983 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:04:20.553748 28780 mvcc.cc:205] Tried to move back new op lower bound from 7333622008010899456 to 7333622007942991872. Current Snapshot: MvccSnapshot[applied={T|T < 7333622008010899456}]
I20260926 15:04:20.554013 28781 mvcc.cc:205] Tried to move back new op lower bound from 7333622008010899456 to 7333622007942991872. Current Snapshot: MvccSnapshot[applied={T|T < 7333622008010899456}]
I20260926 15:04:20.556401 28301 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:04:20.556505 28301 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260926 15:04:20.557395 28301 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 77c04a18d392475298f5d0f3c63f4531: Sending DeleteTablet for 3 replicas of tablet 3b2a7ad845454fae9c8e1330f2546141
I20260926 15:04:20.557768 28424 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b2a7ad845454fae9c8e1330f2546141 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:04:20 UTC) from {username='slave'} at 127.0.0.1:52466
I20260926 15:04:20.557783 28686 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b2a7ad845454fae9c8e1330f2546141 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:04:20 UTC) from {username='slave'} at 127.0.0.1:56932
I20260926 15:04:20.558040 28555 tablet_service.cc:1514] Processing DeleteTablet for tablet 3b2a7ad845454fae9c8e1330f2546141 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-26 15:04:20 UTC) from {username='slave'} at 127.0.0.1:42088
I20260926 15:04:20.558077 28791 tablet_replica.cc:330] T 3b2a7ad845454fae9c8e1330f2546141 P ca151c9f3fe948f9a114c2f99246831a: stopping tablet replica
I20260926 15:04:20.558188 28791 raft_consensus.cc:2229] T 3b2a7ad845454fae9c8e1330f2546141 P ca151c9f3fe948f9a114c2f99246831a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:04:20.558318 28791 raft_consensus.cc:2258] T 3b2a7ad845454fae9c8e1330f2546141 P ca151c9f3fe948f9a114c2f99246831a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:04:20.558575 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 28362
I20260926 15:04:20.560595 28792 tablet_replica.cc:330] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537: stopping tablet replica
I20260926 15:04:20.560716 28792 raft_consensus.cc:2229] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [term 1 LEADER]: Raft consensus shutting down.
I20260926 15:04:20.560851 28792 raft_consensus.cc:2258] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:04:20.560988 28790 tablet_replica.cc:330] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189: stopping tablet replica
I20260926 15:04:20.561097 28790 raft_consensus.cc:2229] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260926 15:04:20.561204 28790 raft_consensus.cc:2258] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260926 15:04:20.561712 28790 ts_tablet_manager.cc:1905] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:04:20.563113 28792 ts_tablet_manager.cc:1905] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260926 15:04:20.563928 28790 ts_tablet_manager.cc:1918] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:04:20.564013 28790 log.cc:1196] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-2/wal/wals/3b2a7ad845454fae9c8e1330f2546141
I20260926 15:04:20.564278 28790 ts_tablet_manager.cc:1939] T 3b2a7ad845454fae9c8e1330f2546141 P 06e6bc4675354394a152c911f53de189: Deleting consensus metadata
W20260926 15:04:20.564913 28287 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.8.102.1:32983 (error 108)
I20260926 15:04:20.565110 28792 ts_tablet_manager.cc:1918] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260926 15:04:20.565150 28286 catalog_manager.cc:4841] TS 06e6bc4675354394a152c911f53de189 (127.8.102.3:38319): tablet 3b2a7ad845454fae9c8e1330f2546141 (table test-table [id=38cba85918f54f74b2aea76326b22803]) successfully deleted
I20260926 15:04:20.565274 28792 log.cc:1196] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537: Deleting WAL directory at /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790434810362722-8600-0/minicluster-data/ts-1/wal/wals/3b2a7ad845454fae9c8e1330f2546141
I20260926 15:04:20.565541 28792 ts_tablet_manager.cc:1939] T 3b2a7ad845454fae9c8e1330f2546141 P 4c1f13c083644b05bb8d258385362537: Deleting consensus metadata
I20260926 15:04:20.565609 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 28493
I20260926 15:04:20.565749 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:20.565617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.1:32983 (local address 127.0.0.1:52470)
0926 15:04:20.565669 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:20.565710 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.1:32983: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 15:04:20.567533 28287 catalog_manager.cc:4584] TS ca151c9f3fe948f9a114c2f99246831a (127.8.102.1:32983): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 3b2a7ad845454fae9c8e1330f2546141: Network error: Client connection negotiation failed: client connection to 127.8.102.1:32983: connect: Connection refused (error 111)
I20260926 15:04:20.569845 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 28624
I20260926 15:04:20.569921 28762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:20.569789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.2:41151 (local address 127.0.0.1:42110)
0926 15:04:20.569853 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:20.569880 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.2:41151: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 15:04:20.574185 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 28271
2026-09-26T15:04:20Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (852 ms)
[----------] 6 tests from EncryptionPolicyTest (8785 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:20 dist-test-slave-jnhs krb5kdc[28796](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:20 dist-test-slave-jnhs krb5kdc[28796](info): set up 2 sockets
Sep 26 15:04:20 dist-test-slave-jnhs krb5kdc[28796](info): commencing operation
krb5kdc: starting...
W20260926 15:04:21.626834 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.033s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:21 dist-test-slave-jnhs krb5kdc[28796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:04:21.754827 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35475
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45155
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35475
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:21.838321 28809 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:21.838547 28809 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:21.838572 28809 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:21.839849 28809 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:21.839900 28809 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:21.839913 28809 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:21.839926 28809 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:21.839938 28809 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:21.841452 28809 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45155
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35475
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35475
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28809
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:21.841661 28809 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:21.841838 28809 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:21.846985 28815 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:21.847011 28817 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:21.846989 28809 server_base.cc:1034] running on GCE node
W20260926 15:04:21.846985 28814 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:21.847519 28809 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:21.847852 28809 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:21.849035 28809 hybrid_clock.cc:648] HybridClock initialized: now 1790435061849024 us; error 48 us; skew 500 ppm
Sep 26 15:04:21 dist-test-slave-jnhs krb5kdc[28796](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435061, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:21.851758 28809 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:04:21.852293 28809 webserver.cc:460] Webserver started at http://127.0.0.1:40017/ using document root <none> and password file <none>
I20260926 15:04:21.852535 28809 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:21.852603 28809 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:21.852757 28809 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:21.853852 28809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "056ae2fcb753430aa8e94c45fc4a5352"
format_stamp: "Formatted at 2026-09-26 15:04:21 on dist-test-slave-jnhs"
server_key: "aaaccbc635cf66061ec68743d3e0bafe"
server_key_iv: "3a8ef309a7a33ce0b67fb0fb97e9b5ae"
server_key_version: "encryptionkey@0"
I20260926 15:04:21.854228 28809 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "056ae2fcb753430aa8e94c45fc4a5352"
format_stamp: "Formatted at 2026-09-26 15:04:21 on dist-test-slave-jnhs"
server_key: "aaaccbc635cf66061ec68743d3e0bafe"
server_key_iv: "3a8ef309a7a33ce0b67fb0fb97e9b5ae"
server_key_version: "encryptionkey@0"
I20260926 15:04:21.855697 28809 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260926 15:04:21.856549 28824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:21.856838 28809 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 15:04:21.856945 28809 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "056ae2fcb753430aa8e94c45fc4a5352"
format_stamp: "Formatted at 2026-09-26 15:04:21 on dist-test-slave-jnhs"
server_key: "aaaccbc635cf66061ec68743d3e0bafe"
server_key_iv: "3a8ef309a7a33ce0b67fb0fb97e9b5ae"
server_key_version: "encryptionkey@0"
I20260926 15:04:21.857023 28809 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:21.868216 28809 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:21.870621 28809 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:21.870869 28809 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:21.874922 28809 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35475
I20260926 15:04:21.874975 28876 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35475 every 8 connection(s)
I20260926 15:04:21.875422 28809 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:21.876430 28877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:21.878866 28877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352: Bootstrap starting.
I20260926 15:04:21.879472 28877 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:21.879766 28877 log.cc:824] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:21.880149 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 28809
I20260926 15:04:21.880280 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:21.880525 8600 external_mini_cluster.cc:1428] Setting key 8086e1ec1fe54c2c34ecad69f9ca90d4
I20260926 15:04:21.880715 28877 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352: No bootstrap required, opened a new log
I20260926 15:04:21.882789 28877 raft_consensus.cc:348] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "056ae2fcb753430aa8e94c45fc4a5352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35475 } }
I20260926 15:04:21.882968 28877 raft_consensus.cc:374] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:21.883016 28877 raft_consensus.cc:729] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 056ae2fcb753430aa8e94c45fc4a5352, State: Initialized, Role: FOLLOWER
I20260926 15:04:21.883153 28877 consensus_queue.cc:260] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "056ae2fcb753430aa8e94c45fc4a5352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35475 } }
I20260926 15:04:21.883255 28877 raft_consensus.cc:388] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:21.883306 28877 raft_consensus.cc:482] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:21.883360 28877 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:21.884284 28877 raft_consensus.cc:504] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "056ae2fcb753430aa8e94c45fc4a5352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35475 } }
I20260926 15:04:21.884416 28877 leader_election.cc:302] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 056ae2fcb753430aa8e94c45fc4a5352; no voters:
I20260926 15:04:21.884640 28877 leader_election.cc:288] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:21.884665 28882 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:21.884793 28882 raft_consensus.cc:686] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352 [term 1 LEADER]: Becoming Leader. State: Replica: 056ae2fcb753430aa8e94c45fc4a5352, State: Running, Role: LEADER
I20260926 15:04:21.884887 28882 consensus_queue.cc:237] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "056ae2fcb753430aa8e94c45fc4a5352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35475 } }
I20260926 15:04:21.884933 28877 sys_catalog.cc:564] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:21.885252 28882 sys_catalog.cc:455] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 056ae2fcb753430aa8e94c45fc4a5352. Latest consensus state: current_term: 1 leader_uuid: "056ae2fcb753430aa8e94c45fc4a5352" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "056ae2fcb753430aa8e94c45fc4a5352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35475 } } }
I20260926 15:04:21.885340 28882 sys_catalog.cc:458] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:21.885267 28883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "056ae2fcb753430aa8e94c45fc4a5352" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "056ae2fcb753430aa8e94c45fc4a5352" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35475 } } }
I20260926 15:04:21.885516 28883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352 [sys.catalog]: This master's current role is: LEADER
W20260926 15:04:21.886925 28897 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:04:21.887051 28897 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:04:21.887130 28887 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:21.887328 28887 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:21.889012 28887 catalog_manager.cc:1347] Generated new cluster ID: a1ede3648b5f4bd7bb67139a6ff5a1fb
I20260926 15:04:21.889070 28887 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 15:04:21 dist-test-slave-jnhs krb5kdc[28796](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435061, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 15:04:21.894224 28881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:21.881138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58639 (local address 127.0.0.1:35475)
0926 15:04:21.881344 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:21.881352 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:21.881378 (+ 26us) server_negotiation.cc:407] Connection header received
0926 15:04:21.882767 (+ 1389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:21.882781 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:21.882841 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:21.882997 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:21.883791 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:21.884865 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:21.885899 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:21.886395 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:21.886933 (+ 538us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:21.889775 (+ 2842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:21.889779 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:21.889779 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:21.889796 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:21.891964 (+ 2168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:21.892471 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:21.892475 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:21.892475 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:21.892621 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:21.892877 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:21.892878 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:21.892878 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:21.893236 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:21.893810 (+ 574us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:21.893904 (+ 94us) server_negotiation.cc:299] Negotiation successful
0926 15:04:21.894016 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142,"thread_start_us":95,"threads_started":1}
I20260926 15:04:21.894521 28887 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:21.895277 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:04:21.895344 28887 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:21.899940 28887 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 056ae2fcb753430aa8e94c45fc4a5352: Generated new TSK 0
I20260926 15:04:21.900133 28887 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:21.906057 28881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:21.897035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41845 (local address 127.0.0.1:35475)
0926 15:04:21.897079 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:21.897081 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:21.898889 (+ 1808us) server_negotiation.cc:407] Connection header received
0926 15:04:21.898943 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:21.898944 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:21.898957 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:21.899029 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:21.899703 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:21.900476 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:21.901578 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:21.901823 (+ 245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:21.901916 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:21.903378 (+ 1462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:21.903379 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:21.903379 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:21.903385 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:21.904903 (+ 1518us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:21.905327 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:21.905327 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:21.905328 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:21.905378 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:21.905606 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:21.905606 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:21.905607 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:21.905724 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:21.905796 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:21.905946 (+ 150us) server_negotiation.cc:299] Negotiation successful
0926 15:04:21.905995 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 15:04:21.908608 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 28809
W20260926 15:04:21.913251 28794 connection.cc:535] client connection to 127.0.0.1:35475 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35475): Connection reset by peer (error 104)
2026-09-26T15:04:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1333 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:21 dist-test-slave-jnhs krb5kdc[28908](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:21 dist-test-slave-jnhs krb5kdc[28908](info): set up 2 sockets
Sep 26 15:04:21 dist-test-slave-jnhs krb5kdc[28908](info): commencing operation
krb5kdc: starting...
W20260926 15:04:22.970993 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:23 dist-test-slave-jnhs krb5kdc[28908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:23Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:04:23.102451 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40695
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40695
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:23.184834 28921 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:23.185097 28921 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:23.185138 28921 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:23.186285 28921 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:23.186357 28921 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:23.186383 28921 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:23.186410 28921 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:23.186442 28921 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:23.187996 28921 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40695
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40695
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28921
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:23.188354 28921 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:23.188591 28921 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 15:04:23.193537 28921 server_base.cc:1034] running on GCE node
W20260926 15:04:23.193497 28929 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:23.193522 28927 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:23.193522 28926 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:23.193990 28921 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:23.194255 28921 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:23.195446 28921 hybrid_clock.cc:648] HybridClock initialized: now 1790435063195429 us; error 33 us; skew 500 ppm
Sep 26 15:04:23 dist-test-slave-jnhs krb5kdc[28908](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:23.198240 28921 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:04:23.198789 28921 webserver.cc:460] Webserver started at http://127.0.0.1:37685/ using document root <none> and password file <none>
I20260926 15:04:23.199028 28921 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:23.199075 28921 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:23.199218 28921 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:23.200294 28921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "a8ac4b48ff5644088bb20578bbfb0bc1"
format_stamp: "Formatted at 2026-09-26 15:04:23 on dist-test-slave-jnhs"
server_key: "03e4894ef0c99ea3b41ec08c6fc60dfc"
server_key_iv: "1ccd18b0dc2f976a8356609e5618dc20"
server_key_version: "encryptionkey@0"
I20260926 15:04:23.200654 28921 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "a8ac4b48ff5644088bb20578bbfb0bc1"
format_stamp: "Formatted at 2026-09-26 15:04:23 on dist-test-slave-jnhs"
server_key: "03e4894ef0c99ea3b41ec08c6fc60dfc"
server_key_iv: "1ccd18b0dc2f976a8356609e5618dc20"
server_key_version: "encryptionkey@0"
I20260926 15:04:23.202000 28921 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:04:23.202889 28936 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:23.203176 28921 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:23.203269 28921 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "a8ac4b48ff5644088bb20578bbfb0bc1"
format_stamp: "Formatted at 2026-09-26 15:04:23 on dist-test-slave-jnhs"
server_key: "03e4894ef0c99ea3b41ec08c6fc60dfc"
server_key_iv: "1ccd18b0dc2f976a8356609e5618dc20"
server_key_version: "encryptionkey@0"
I20260926 15:04:23.203339 28921 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:23.217952 28921 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:23.220249 28921 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:23.220409 28921 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:23.225400 28921 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40695
I20260926 15:04:23.225473 28988 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40695 every 8 connection(s)
I20260926 15:04:23.225814 28921 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:23.227053 28989 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:23.227732 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 28921
I20260926 15:04:23.227849 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:23.228058 8600 external_mini_cluster.cc:1428] Setting key 29cea364dae3b4899e34eaa645ec27d6
I20260926 15:04:23.230041 28989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1: Bootstrap starting.
I20260926 15:04:23.230724 28989 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:23.231035 28989 log.cc:824] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:23.231784 28989 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1: No bootstrap required, opened a new log
I20260926 15:04:23.233639 28992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:23.228611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47651 (local address 127.0.0.1:40695)
0926 15:04:23.228796 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:23.228804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:23.228830 (+ 26us) server_negotiation.cc:407] Connection header received
0926 15:04:23.229238 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:23.229244 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:23.229290 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:23.229405 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:23.229635 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:23.230573 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:23.231655 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:23.232148 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:23.232234 (+ 86us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:23.232969 (+ 735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:23.232973 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:23.232974 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:23.233150 (+ 176us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:23.233180 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:23.233191 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:23.233256 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:23.233377 (+ 121us) server_negotiation.cc:299] Negotiation successful
0926 15:04:23.233425 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":94,"threads_started":1}
I20260926 15:04:23.234182 28989 raft_consensus.cc:348] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ac4b48ff5644088bb20578bbfb0bc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } }
I20260926 15:04:23.234313 28989 raft_consensus.cc:374] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:23.234375 28989 raft_consensus.cc:729] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a8ac4b48ff5644088bb20578bbfb0bc1, State: Initialized, Role: FOLLOWER
I20260926 15:04:23.234498 28989 consensus_queue.cc:260] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ac4b48ff5644088bb20578bbfb0bc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } }
I20260926 15:04:23.234602 28989 raft_consensus.cc:388] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:23.234683 28989 raft_consensus.cc:482] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:23.234728 28989 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:23.234860 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:04:23.235357 28989 raft_consensus.cc:504] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ac4b48ff5644088bb20578bbfb0bc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } }
I20260926 15:04:23.235471 28989 leader_election.cc:302] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a8ac4b48ff5644088bb20578bbfb0bc1; no voters:
I20260926 15:04:23.235680 28989 leader_election.cc:288] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:23.235720 28994 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:23.235849 28994 raft_consensus.cc:686] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1 [term 1 LEADER]: Becoming Leader. State: Replica: a8ac4b48ff5644088bb20578bbfb0bc1, State: Running, Role: LEADER
I20260926 15:04:23.235950 28989 sys_catalog.cc:564] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:23.235930 28994 consensus_queue.cc:237] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ac4b48ff5644088bb20578bbfb0bc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } }
I20260926 15:04:23.237393 28994 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a8ac4b48ff5644088bb20578bbfb0bc1. Latest consensus state: current_term: 1 leader_uuid: "a8ac4b48ff5644088bb20578bbfb0bc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ac4b48ff5644088bb20578bbfb0bc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } } }
I20260926 15:04:23.237488 28994 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:23.237628 28997 sys_catalog.cc:455] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a8ac4b48ff5644088bb20578bbfb0bc1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a8ac4b48ff5644088bb20578bbfb0bc1" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40695 } } }
I20260926 15:04:23.237694 29014 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:23.237731 28997 sys_catalog.cc:458] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:23.238224 29014 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:23.239974 29014 catalog_manager.cc:1347] Generated new cluster ID: b212f9da4e93459492e8fb5a1a054add
I20260926 15:04:23.240032 29014 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:23.241251 28992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:23.236383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59437 (local address 127.0.0.1:40695)
0926 15:04:23.236424 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:23.236426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:23.237785 (+ 1359us) server_negotiation.cc:407] Connection header received
0926 15:04:23.237968 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:23.237969 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:23.237980 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:23.238009 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:23.238240 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:23.238965 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:23.239964 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:23.240248 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:23.240283 (+ 35us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:23.240962 (+ 679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:23.240963 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:23.240964 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:23.240968 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:23.240983 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:23.240987 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:23.241084 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:23.241145 (+ 61us) server_negotiation.cc:299] Negotiation successful
0926 15:04:23.241155 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 15:04:23.251288 29014 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:23.252022 29014 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:23.259007 29014 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a8ac4b48ff5644088bb20578bbfb0bc1: Generated new TSK 0
I20260926 15:04:23.259219 29014 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:23.261512 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 28921
W20260926 15:04:23.265606 28906 connection.cc:535] client connection to 127.0.0.1:40695 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40695): Connection reset by peer (error 104)
2026-09-26T15:04:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:23 dist-test-slave-jnhs krb5kdc[29020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:23 dist-test-slave-jnhs krb5kdc[29020](info): set up 2 sockets
Sep 26 15:04:23 dist-test-slave-jnhs krb5kdc[29020](info): commencing operation
krb5kdc: starting...
W20260926 15:04:24.318980 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.039s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:24 dist-test-slave-jnhs krb5kdc[29020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435064, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:04:24.446877 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46661
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46661
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:24.535000 29033 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:24.535290 29033 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:24.535336 29033 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:24.536604 29033 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:24.536700 29033 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:24.536739 29033 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:24.536773 29033 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:24.536796 29033 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:24.538381 29033 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46661
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46661
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29033
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:24.538721 29033 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:24.538977 29033 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:24.544204 29039 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:24.544361 29038 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:24.544409 29041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:24.544298 29033 server_base.cc:1034] running on GCE node
I20260926 15:04:24.544726 29033 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:24.545073 29033 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:24.546254 29033 hybrid_clock.cc:648] HybridClock initialized: now 1790435064546216 us; error 54 us; skew 500 ppm
Sep 26 15:04:24 dist-test-slave-jnhs krb5kdc[29020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:24.549135 29033 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:04:24.549734 29033 webserver.cc:460] Webserver started at http://127.0.0.1:33745/ using document root <none> and password file <none>
I20260926 15:04:24.549994 29033 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:24.550072 29033 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:24.550243 29033 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:24.551508 29033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "848a05c538a145b8b3ce07d4c45006e4"
format_stamp: "Formatted at 2026-09-26 15:04:24 on dist-test-slave-jnhs"
server_key: "1c1e34c2af7ed7bc76a95b63dad0b173"
server_key_iv: "9d909bec9c8f433ead356d01efc80c59"
server_key_version: "encryptionkey@0"
I20260926 15:04:24.551909 29033 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "848a05c538a145b8b3ce07d4c45006e4"
format_stamp: "Formatted at 2026-09-26 15:04:24 on dist-test-slave-jnhs"
server_key: "1c1e34c2af7ed7bc76a95b63dad0b173"
server_key_iv: "9d909bec9c8f433ead356d01efc80c59"
server_key_version: "encryptionkey@0"
I20260926 15:04:24.553452 29033 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 15:04:24.554352 29048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:24.554561 29033 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:04:24.554713 29033 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "848a05c538a145b8b3ce07d4c45006e4"
format_stamp: "Formatted at 2026-09-26 15:04:24 on dist-test-slave-jnhs"
server_key: "1c1e34c2af7ed7bc76a95b63dad0b173"
server_key_iv: "9d909bec9c8f433ead356d01efc80c59"
server_key_version: "encryptionkey@0"
I20260926 15:04:24.554803 29033 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:24.558529 29033 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:24.560756 29033 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:24.560936 29033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:24.565435 29033 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46661
I20260926 15:04:24.565469 29100 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46661 every 8 connection(s)
I20260926 15:04:24.565842 29033 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:24.566983 29101 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:24.569301 29101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4: Bootstrap starting.
I20260926 15:04:24.569855 29101 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:24.570119 29101 log.cc:824] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:24.570899 29101 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4: No bootstrap required, opened a new log
I20260926 15:04:24.572330 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 29033
I20260926 15:04:24.572449 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:24.572649 8600 external_mini_cluster.cc:1428] Setting key 36341ee88554fd965c837149f0fa9b59
I20260926 15:04:24.572911 29101 raft_consensus.cc:348] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848a05c538a145b8b3ce07d4c45006e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46661 } }
I20260926 15:04:24.573132 29101 raft_consensus.cc:374] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:24.573194 29101 raft_consensus.cc:729] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 848a05c538a145b8b3ce07d4c45006e4, State: Initialized, Role: FOLLOWER
I20260926 15:04:24.573345 29101 consensus_queue.cc:260] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848a05c538a145b8b3ce07d4c45006e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46661 } }
I20260926 15:04:24.573454 29101 raft_consensus.cc:388] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:24.573482 29101 raft_consensus.cc:482] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:24.573562 29101 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:24.574918 29101 raft_consensus.cc:504] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848a05c538a145b8b3ce07d4c45006e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46661 } }
I20260926 15:04:24.575059 29101 leader_election.cc:302] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 848a05c538a145b8b3ce07d4c45006e4; no voters:
I20260926 15:04:24.575213 29105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:24.573214 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57305 (local address 127.0.0.1:46661)
0926 15:04:24.573462 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:24.573471 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:24.573556 (+ 85us) server_negotiation.cc:407] Connection header received
0926 15:04:24.573679 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:24.573685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:24.573707 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:24.573821 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:24.574574 (+ 753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:24.574577 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:24.574577 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:24.574581 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:24.574622 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:24.574688 (+ 66us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:24.574755 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:24.574848 (+ 93us) server_negotiation.cc:299] Negotiation successful
0926 15:04:24.574915 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":168,"thread_start_us":134,"threads_started":1}
I20260926 15:04:24.575223 29101 leader_election.cc:288] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:24.575575 29101 sys_catalog.cc:564] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:24.575277 29106 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:24.577488 29106 raft_consensus.cc:686] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4 [term 1 LEADER]: Becoming Leader. State: Replica: 848a05c538a145b8b3ce07d4c45006e4, State: Running, Role: LEADER
I20260926 15:04:24.577682 29106 consensus_queue.cc:237] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848a05c538a145b8b3ce07d4c45006e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46661 } }
I20260926 15:04:24.578125 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:04:24.578678 29107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "848a05c538a145b8b3ce07d4c45006e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848a05c538a145b8b3ce07d4c45006e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46661 } } }
I20260926 15:04:24.578774 29107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:24.578960 29110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 848a05c538a145b8b3ce07d4c45006e4. Latest consensus state: current_term: 1 leader_uuid: "848a05c538a145b8b3ce07d4c45006e4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "848a05c538a145b8b3ce07d4c45006e4" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46661 } } }
W20260926 15:04:24.579018 29120 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260926 15:04:24.579036 29110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4 [sys.catalog]: This master's current role is: LEADER
W20260926 15:04:24.579053 29120 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:04:24.579207 29122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:24.579396 29122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:24.581127 29122 catalog_manager.cc:1347] Generated new cluster ID: 5215d7b6c1454af48aee6bfa3ee0dd25
I20260926 15:04:24.581203 29122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:24.581416 29105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:24.579957 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55011 (local address 127.0.0.1:46661)
0926 15:04:24.579991 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:24.579993 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:24.580288 (+ 295us) server_negotiation.cc:407] Connection header received
0926 15:04:24.580390 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:24.580391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:24.580409 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:24.580438 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:24.581176 (+ 738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:24.581177 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:24.581177 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:24.581182 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:24.581198 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:24.581202 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:24.581234 (+ 32us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:24.581341 (+ 107us) server_negotiation.cc:299] Negotiation successful
0926 15:04:24.581356 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 15:04:24.589102 29122 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:24.589298 29122 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:24.592796 29122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 848a05c538a145b8b3ce07d4c45006e4: Generated new TSK 0
I20260926 15:04:24.593065 29122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:24.596169 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 29033
2026-09-26T15:04:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:24 dist-test-slave-jnhs krb5kdc[29132](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:24 dist-test-slave-jnhs krb5kdc[29132](info): set up 2 sockets
Sep 26 15:04:24 dist-test-slave-jnhs krb5kdc[29132](info): commencing operation
krb5kdc: starting...
W20260926 15:04:25.674913 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.060s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:25 dist-test-slave-jnhs krb5kdc[29132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:04:25.802501 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43075
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39803
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43075
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:25.889770 29145 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:25.890029 29145 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:25.890079 29145 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:25.891355 29145 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:25.891423 29145 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:25.891449 29145 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:25.891479 29145 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:25.891510 29145 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:25.893069 29145 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39803
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43075
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29145
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:25.893370 29145 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:25.893610 29145 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:25.898573 29153 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:25.898626 29145 server_base.cc:1034] running on GCE node
W20260926 15:04:25.898614 29151 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:25.898625 29150 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:25.899068 29145 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:25.899398 29145 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:25.900588 29145 hybrid_clock.cc:648] HybridClock initialized: now 1790435065900565 us; error 38 us; skew 500 ppm
Sep 26 15:04:25 dist-test-slave-jnhs krb5kdc[29132](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435065, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:25.903285 29145 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:04:25.903702 29145 webserver.cc:460] Webserver started at http://127.0.0.1:34631/ using document root <none> and password file <none>
I20260926 15:04:25.903899 29145 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:25.903942 29145 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:25.904032 29145 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:25.905077 29145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "4da9b1915326429687ed273f17471336"
format_stamp: "Formatted at 2026-09-26 15:04:25 on dist-test-slave-jnhs"
server_key: "69c8eebb4b67385d10fabcfcaed4f18b"
server_key_iv: "318cc2a013335dedb42683486a1c9146"
server_key_version: "encryptionkey@0"
I20260926 15:04:25.905380 29145 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "4da9b1915326429687ed273f17471336"
format_stamp: "Formatted at 2026-09-26 15:04:25 on dist-test-slave-jnhs"
server_key: "69c8eebb4b67385d10fabcfcaed4f18b"
server_key_iv: "318cc2a013335dedb42683486a1c9146"
server_key_version: "encryptionkey@0"
I20260926 15:04:25.906785 29145 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 15:04:25.907642 29160 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:25.907837 29145 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:25.907927 29145 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "4da9b1915326429687ed273f17471336"
format_stamp: "Formatted at 2026-09-26 15:04:25 on dist-test-slave-jnhs"
server_key: "69c8eebb4b67385d10fabcfcaed4f18b"
server_key_iv: "318cc2a013335dedb42683486a1c9146"
server_key_version: "encryptionkey@0"
I20260926 15:04:25.907994 29145 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:25.937592 29145 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:25.939953 29145 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:25.940084 29145 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:25.943974 29145 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43075
I20260926 15:04:25.944047 29212 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43075 every 8 connection(s)
I20260926 15:04:25.944414 29145 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:25.945545 29213 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:25.948006 29213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336: Bootstrap starting.
I20260926 15:04:25.948459 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 29145
I20260926 15:04:25.948577 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:25.948774 8600 external_mini_cluster.cc:1428] Setting key 43e2c491614d12773ad096d684fedba1
I20260926 15:04:25.948776 29213 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:25.949081 29213 log.cc:824] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:25.949846 29213 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336: No bootstrap required, opened a new log
I20260926 15:04:25.951761 29213 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da9b1915326429687ed273f17471336" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43075 } }
I20260926 15:04:25.951903 29213 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:25.951925 29213 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4da9b1915326429687ed273f17471336, State: Initialized, Role: FOLLOWER
I20260926 15:04:25.952070 29213 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da9b1915326429687ed273f17471336" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43075 } }
I20260926 15:04:25.952157 29213 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:25.952198 29213 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:25.952241 29213 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:25.952818 29213 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da9b1915326429687ed273f17471336" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43075 } }
I20260926 15:04:25.952947 29213 leader_election.cc:302] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4da9b1915326429687ed273f17471336; no voters:
I20260926 15:04:25.953150 29213 leader_election.cc:288] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:25.953176 29218 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:25.953299 29218 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336 [term 1 LEADER]: Becoming Leader. State: Replica: 4da9b1915326429687ed273f17471336, State: Running, Role: LEADER
I20260926 15:04:25.953429 29213 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:25.953437 29218 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da9b1915326429687ed273f17471336" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43075 } }
I20260926 15:04:25.953894 29219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4da9b1915326429687ed273f17471336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da9b1915326429687ed273f17471336" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43075 } } }
I20260926 15:04:25.953909 29220 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4da9b1915326429687ed273f17471336. Latest consensus state: current_term: 1 leader_uuid: "4da9b1915326429687ed273f17471336" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4da9b1915326429687ed273f17471336" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43075 } } }
I20260926 15:04:25.954030 29219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:25.954054 29220 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336 [sys.catalog]: This master's current role is: LEADER
W20260926 15:04:25.955664 29234 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:04:25.955711 29234 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Sep 26 15:04:25 dist-test-slave-jnhs krb5kdc[29132](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435065, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 15:04:25.956185 29222 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:25.956389 29222 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:25.958030 29222 catalog_manager.cc:1347] Generated new cluster ID: 557e3171d6454311a7b1dff0b1743cae
I20260926 15:04:25.958158 29222 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:25.960685 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:25.949359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40577 (local address 127.0.0.1:43075)
0926 15:04:25.949583 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:25.949591 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:25.949617 (+ 26us) server_negotiation.cc:407] Connection header received
0926 15:04:25.950119 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:25.950127 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:25.950177 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:25.950325 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:25.951060 (+ 735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:25.952060 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:25.952892 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:25.953505 (+ 613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:25.953964 (+ 459us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:25.956545 (+ 2581us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:25.956550 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:25.956550 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:25.956563 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:25.958954 (+ 2391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:25.959373 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:25.959376 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:25.959377 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:25.959485 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:25.959687 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:25.959688 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:25.959688 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:25.960000 (+ 312us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:25.960194 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:25.960317 (+ 123us) server_negotiation.cc:299] Negotiation successful
0926 15:04:25.960461 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
I20260926 15:04:25.961778 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:04:25.970901 29222 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:25.971437 29222 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:25.971561 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:25.963521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38861 (local address 127.0.0.1:43075)
0926 15:04:25.963559 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:25.963561 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:25.963587 (+ 26us) server_negotiation.cc:407] Connection header received
0926 15:04:25.963663 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:25.963664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:25.963672 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:25.963723 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:25.964388 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:25.965137 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:25.966222 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:25.966747 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:25.966831 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:25.968354 (+ 1523us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:25.968355 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:25.968356 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:25.968363 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:25.970029 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:25.970467 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:25.970468 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:25.970469 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:25.970539 (+ 70us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:25.970828 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:25.970829 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:25.970829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:25.970972 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:25.971082 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:25.971289 (+ 207us) server_negotiation.cc:299] Negotiation successful
0926 15:04:25.971379 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 15:04:25.977872 29222 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4da9b1915326429687ed273f17471336: Generated new TSK 0
I20260926 15:04:25.978101 29222 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:25.981122 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 29145
2026-09-26T15:04:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:26 dist-test-slave-jnhs krb5kdc[29244](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:26 dist-test-slave-jnhs krb5kdc[29244](info): set up 2 sockets
Sep 26 15:04:26 dist-test-slave-jnhs krb5kdc[29244](info): commencing operation
krb5kdc: starting...
W20260926 15:04:27.042951 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.006s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:27 dist-test-slave-jnhs krb5kdc[29244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435067, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:04:27.170996 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:43077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:27.255136 29257 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:27.255349 29257 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:27.255374 29257 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:27.256542 29257 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:27.256585 29257 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:27.256599 29257 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:27.256620 29257 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:27.256632 29257 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:27.258124 29257 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29257
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:27.258338 29257 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:27.258515 29257 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:27.263496 29262 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:27.263528 29265 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:27.263496 29263 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:27.263901 29257 server_base.cc:1034] running on GCE node
I20260926 15:04:27.264178 29257 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:27.264504 29257 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:27.265690 29257 hybrid_clock.cc:648] HybridClock initialized: now 1790435067265670 us; error 41 us; skew 500 ppm
Sep 26 15:04:27 dist-test-slave-jnhs krb5kdc[29244](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435067, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:27.268462 29257 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:04:27.268986 29257 webserver.cc:460] Webserver started at http://127.0.0.1:43803/ using document root <none> and password file <none>
I20260926 15:04:27.269251 29257 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:27.269322 29257 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:27.269474 29257 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:27.270571 29257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "95227583a9fc462c92cfc3de235b2e8a"
format_stamp: "Formatted at 2026-09-26 15:04:27 on dist-test-slave-jnhs"
server_key: "6f3733386dc3cb185f03a8881a59d6ce"
server_key_iv: "d88064b9d4a0d83def718b4100c6c8c5"
server_key_version: "encryptionkey@0"
I20260926 15:04:27.271006 29257 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "95227583a9fc462c92cfc3de235b2e8a"
format_stamp: "Formatted at 2026-09-26 15:04:27 on dist-test-slave-jnhs"
server_key: "6f3733386dc3cb185f03a8881a59d6ce"
server_key_iv: "d88064b9d4a0d83def718b4100c6c8c5"
server_key_version: "encryptionkey@0"
I20260926 15:04:27.272382 29257 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:27.273206 29272 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:27.273465 29257 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:27.273587 29257 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "95227583a9fc462c92cfc3de235b2e8a"
format_stamp: "Formatted at 2026-09-26 15:04:27 on dist-test-slave-jnhs"
server_key: "6f3733386dc3cb185f03a8881a59d6ce"
server_key_iv: "d88064b9d4a0d83def718b4100c6c8c5"
server_key_version: "encryptionkey@0"
I20260926 15:04:27.273685 29257 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:27.284730 29257 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:27.287216 29257 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:27.287393 29257 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:27.291746 29257 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34053
I20260926 15:04:27.291781 29324 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34053 every 8 connection(s)
I20260926 15:04:27.292161 29257 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:27.293186 29325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:27.295682 29325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a: Bootstrap starting.
I20260926 15:04:27.296260 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 29257
I20260926 15:04:27.296406 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:27.296528 29325 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:27.296638 8600 external_mini_cluster.cc:1428] Setting key 451d191247e9e132752982a23073fce4
I20260926 15:04:27.296868 29325 log.cc:824] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:27.297695 29325 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a: No bootstrap required, opened a new log
I20260926 15:04:27.299536 29325 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95227583a9fc462c92cfc3de235b2e8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } }
I20260926 15:04:27.299717 29325 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:27.299765 29325 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95227583a9fc462c92cfc3de235b2e8a, State: Initialized, Role: FOLLOWER
I20260926 15:04:27.299901 29325 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95227583a9fc462c92cfc3de235b2e8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } }
I20260926 15:04:27.300004 29325 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:27.300046 29325 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:27.300107 29325 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:27.300714 29325 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95227583a9fc462c92cfc3de235b2e8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } }
I20260926 15:04:27.300803 29325 leader_election.cc:302] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95227583a9fc462c92cfc3de235b2e8a; no voters:
I20260926 15:04:27.301014 29325 leader_election.cc:288] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:27.301048 29330 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:27.301159 29330 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a [term 1 LEADER]: Becoming Leader. State: Replica: 95227583a9fc462c92cfc3de235b2e8a, State: Running, Role: LEADER
I20260926 15:04:27.301306 29325 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:27.301384 29330 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95227583a9fc462c92cfc3de235b2e8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } }
I20260926 15:04:27.302959 29331 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95227583a9fc462c92cfc3de235b2e8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95227583a9fc462c92cfc3de235b2e8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } } }
I20260926 15:04:27.302968 29332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95227583a9fc462c92cfc3de235b2e8a. Latest consensus state: current_term: 1 leader_uuid: "95227583a9fc462c92cfc3de235b2e8a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95227583a9fc462c92cfc3de235b2e8a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34053 } } }
I20260926 15:04:27.303138 29331 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:27.303188 29332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a [sys.catalog]: This master's current role is: LEADER
W20260926 15:04:27.303254 29345 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:04:27.303335 29345 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:04:27.303735 29347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:27.303980 29347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:27.305104 29329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:27.297228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52903 (local address 127.0.0.1:34053)
0926 15:04:27.297431 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:27.297441 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:27.297480 (+ 39us) server_negotiation.cc:407] Connection header received
0926 15:04:27.299547 (+ 2067us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:27.299558 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:27.299637 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:27.299811 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:27.300099 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:27.301577 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:27.302663 (+ 1086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:27.303321 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:27.303435 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:27.304282 (+ 847us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:27.304286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:27.304286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:27.304543 (+ 257us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:27.304596 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:27.304613 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:27.304688 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:27.304818 (+ 130us) server_negotiation.cc:299] Negotiation successful
0926 15:04:27.304888 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":116,"threads_started":1}
I20260926 15:04:27.305835 29347 catalog_manager.cc:1347] Generated new cluster ID: da2e9b6f2dd54044a6c0f38508e43917
I20260926 15:04:27.305889 29347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:27.306116 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:04:27.314450 29329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:27.310948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52799 (local address 127.0.0.1:34053)
0926 15:04:27.310987 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:27.310988 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:27.311048 (+ 60us) server_negotiation.cc:407] Connection header received
0926 15:04:27.311106 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:27.311106 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:27.311125 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:27.311192 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:27.311328 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:27.312048 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:27.313064 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:27.313424 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:27.313500 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:27.314190 (+ 690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:27.314191 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:27.314192 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:27.314201 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:27.314219 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:27.314223 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:27.314284 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:27.314338 (+ 54us) server_negotiation.cc:299] Negotiation successful
0926 15:04:27.314365 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 15:04:27.328774 29347 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:27.329355 29347 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:27.337258 29347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95227583a9fc462c92cfc3de235b2e8a: Generated new TSK 0
I20260926 15:04:27.337486 29347 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:27.345027 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 29257
2026-09-26T15:04:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:27 dist-test-slave-jnhs krb5kdc[29356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:27 dist-test-slave-jnhs krb5kdc[29356](info): set up 2 sockets
Sep 26 15:04:27 dist-test-slave-jnhs krb5kdc[29356](info): commencing operation
krb5kdc: starting...
W20260926 15:04:28.411011 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.048s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:28 dist-test-slave-jnhs krb5kdc[29356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:04:28.542598 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39703
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37389
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39703
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:28.626770 29369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:28.627049 29369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:28.627095 29369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:28.628434 29369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:28.628526 29369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:28.628551 29369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:28.628573 29369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:28.628603 29369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:28.630299 29369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37389
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39703
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39703
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29369
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:28.630622 29369 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:28.630891 29369 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:28.636240 29377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:28.636296 29369 server_base.cc:1034] running on GCE node
W20260926 15:04:28.636174 29374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:28.636186 29375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:28.636673 29369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:28.637008 29369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:28.638259 29369 hybrid_clock.cc:648] HybridClock initialized: now 1790435068638219 us; error 53 us; skew 500 ppm
Sep 26 15:04:28 dist-test-slave-jnhs krb5kdc[29356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435068, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:28.645529 29369 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:04:28.646129 29369 webserver.cc:460] Webserver started at http://127.0.0.1:43791/ using document root <none> and password file <none>
I20260926 15:04:28.646430 29369 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:28.646512 29369 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:28.646740 29369 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:28.648106 29369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "7ab39049e340413595ac61d14274cd9e"
format_stamp: "Formatted at 2026-09-26 15:04:28 on dist-test-slave-jnhs"
server_key: "d9744f55fd573824dd400fc5b8a8b9d2"
server_key_iv: "12e5a803006af93835d0dd67cbb3ffa2"
server_key_version: "encryptionkey@0"
I20260926 15:04:28.648519 29369 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "7ab39049e340413595ac61d14274cd9e"
format_stamp: "Formatted at 2026-09-26 15:04:28 on dist-test-slave-jnhs"
server_key: "d9744f55fd573824dd400fc5b8a8b9d2"
server_key_iv: "12e5a803006af93835d0dd67cbb3ffa2"
server_key_version: "encryptionkey@0"
I20260926 15:04:28.650012 29369 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:04:28.651180 29384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:28.651530 29369 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:28.651674 29369 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "7ab39049e340413595ac61d14274cd9e"
format_stamp: "Formatted at 2026-09-26 15:04:28 on dist-test-slave-jnhs"
server_key: "d9744f55fd573824dd400fc5b8a8b9d2"
server_key_iv: "12e5a803006af93835d0dd67cbb3ffa2"
server_key_version: "encryptionkey@0"
I20260926 15:04:28.651787 29369 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:28.655716 29369 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:28.658207 29369 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:28.658391 29369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:28.663049 29369 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39703
I20260926 15:04:28.663093 29436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39703 every 8 connection(s)
I20260926 15:04:28.663514 29369 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:28.664551 29437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:28.667084 29437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e: Bootstrap starting.
I20260926 15:04:28.667654 29437 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:28.667900 29437 log.cc:824] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:28.668094 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 29369
I20260926 15:04:28.668200 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:28.668340 8600 external_mini_cluster.cc:1428] Setting key f35e657fd77d120ef76a25ef928293f8
I20260926 15:04:28.668869 29437 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e: No bootstrap required, opened a new log
I20260926 15:04:28.671366 29437 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab39049e340413595ac61d14274cd9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39703 } }
I20260926 15:04:28.671572 29437 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:28.671614 29437 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ab39049e340413595ac61d14274cd9e, State: Initialized, Role: FOLLOWER
I20260926 15:04:28.671818 29437 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab39049e340413595ac61d14274cd9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39703 } }
I20260926 15:04:28.671936 29437 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:28.671983 29437 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:28.672041 29437 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:28.672183 29441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:28.668963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47515 (local address 127.0.0.1:39703)
0926 15:04:28.669179 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:28.669189 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:28.669223 (+ 34us) server_negotiation.cc:407] Connection header received
0926 15:04:28.670252 (+ 1029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:28.670263 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:28.670303 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:28.670478 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:28.671382 (+ 904us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:28.671386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:28.671386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:28.671391 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:28.671439 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:28.671453 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:28.671542 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:28.671722 (+ 180us) server_negotiation.cc:299] Negotiation successful
0926 15:04:28.671776 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":106,"threads_started":1}
I20260926 15:04:28.672883 29437 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab39049e340413595ac61d14274cd9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39703 } }
I20260926 15:04:28.673009 29437 leader_election.cc:302] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ab39049e340413595ac61d14274cd9e; no voters:
I20260926 15:04:28.673200 29437 leader_election.cc:288] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:28.673256 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:04:28.673257 29442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:28.673414 29442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e [term 1 LEADER]: Becoming Leader. State: Replica: 7ab39049e340413595ac61d14274cd9e, State: Running, Role: LEADER
I20260926 15:04:28.673477 29437 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:28.673560 29442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab39049e340413595ac61d14274cd9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39703 } }
I20260926 15:04:28.674005 29442 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ab39049e340413595ac61d14274cd9e. Latest consensus state: current_term: 1 leader_uuid: "7ab39049e340413595ac61d14274cd9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab39049e340413595ac61d14274cd9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39703 } } }
I20260926 15:04:28.674163 29442 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:28.674017 29444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ab39049e340413595ac61d14274cd9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ab39049e340413595ac61d14274cd9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39703 } } }
I20260926 15:04:28.674279 29444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:28.675077 29455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:28.675745 29455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:28.677265 29441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:28.675142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60079 (local address 127.0.0.1:39703)
0926 15:04:28.675179 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:28.675181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:28.675472 (+ 291us) server_negotiation.cc:407] Connection header received
0926 15:04:28.675627 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:28.675628 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:28.675634 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:28.675704 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:28.676639 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:28.676641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:28.676641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:28.676646 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:28.676663 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:28.676668 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:28.676721 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:28.677176 (+ 455us) server_negotiation.cc:299] Negotiation successful
0926 15:04:28.677191 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 15:04:28.678063 29455 catalog_manager.cc:1347] Generated new cluster ID: 2d44170dd7a549c0af7aad973a9bc00d
I20260926 15:04:28.678141 29455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:28.691051 29455 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:28.691284 29455 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:28.698196 29455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ab39049e340413595ac61d14274cd9e: Generated new TSK 0
I20260926 15:04:28.698405 29455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:28.701684 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 29369
2026-09-26T15:04:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1357 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:28 dist-test-slave-jnhs krb5kdc[29468](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:28 dist-test-slave-jnhs krb5kdc[29468](info): set up 2 sockets
Sep 26 15:04:28 dist-test-slave-jnhs krb5kdc[29468](info): commencing operation
krb5kdc: starting...
W20260926 15:04:29.767380 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.047s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:29 dist-test-slave-jnhs krb5kdc[29468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:04:29.898273 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38897
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38897
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:29.983045 29481 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:29.983335 29481 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:29.983377 29481 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:29.984580 29481 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:29.984661 29481 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:29.984685 29481 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:29.984727 29481 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:29.984751 29481 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:29.986282 29481 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29481
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:29.986575 29481 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:29.986865 29481 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:29.991989 29489 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:29.991995 29486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:29.991995 29487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:29.992036 29481 server_base.cc:1034] running on GCE node
I20260926 15:04:29.992587 29481 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:29.992933 29481 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:29.994125 29481 hybrid_clock.cc:648] HybridClock initialized: now 1790435069994082 us; error 61 us; skew 500 ppm
Sep 26 15:04:29 dist-test-slave-jnhs krb5kdc[29468](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:29.997048 29481 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:04:29.997576 29481 webserver.cc:460] Webserver started at http://127.0.0.1:43953/ using document root <none> and password file <none>
I20260926 15:04:29.997826 29481 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:29.997900 29481 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:29.998068 29481 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:29.999332 29481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "4f4aea9b0a18459cb8cea0887b296a84"
format_stamp: "Formatted at 2026-09-26 15:04:29 on dist-test-slave-jnhs"
server_key: "7be3317aea3a33d69e324d293af90b33"
server_key_iv: "066c9903083780a8f8f35e64f1433479"
server_key_version: "encryptionkey@0"
I20260926 15:04:29.999713 29481 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "4f4aea9b0a18459cb8cea0887b296a84"
format_stamp: "Formatted at 2026-09-26 15:04:29 on dist-test-slave-jnhs"
server_key: "7be3317aea3a33d69e324d293af90b33"
server_key_iv: "066c9903083780a8f8f35e64f1433479"
server_key_version: "encryptionkey@0"
I20260926 15:04:30.001415 29481 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:04:30.002295 29496 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:30.002545 29481 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:30.002687 29481 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "4f4aea9b0a18459cb8cea0887b296a84"
format_stamp: "Formatted at 2026-09-26 15:04:29 on dist-test-slave-jnhs"
server_key: "7be3317aea3a33d69e324d293af90b33"
server_key_iv: "066c9903083780a8f8f35e64f1433479"
server_key_version: "encryptionkey@0"
I20260926 15:04:30.002782 29481 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:30.012084 29481 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:30.014797 29481 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:30.014930 29481 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:30.019111 29481 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38897
I20260926 15:04:30.019129 29548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38897 every 8 connection(s)
I20260926 15:04:30.019582 29481 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:30.020790 29549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:30.023296 29549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84: Bootstrap starting.
I20260926 15:04:30.023552 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 29481
I20260926 15:04:30.023682 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:30.023881 8600 external_mini_cluster.cc:1428] Setting key 51c91b50c01019fcb418670310d32119
I20260926 15:04:30.024081 29549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:30.024497 29549 log.cc:824] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:30.025342 29549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84: No bootstrap required, opened a new log
I20260926 15:04:30.027940 29549 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4aea9b0a18459cb8cea0887b296a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38897 } }
I20260926 15:04:30.028146 29549 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:30.028201 29549 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4f4aea9b0a18459cb8cea0887b296a84, State: Initialized, Role: FOLLOWER
I20260926 15:04:30.028347 29549 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4aea9b0a18459cb8cea0887b296a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38897 } }
I20260926 15:04:30.028440 29549 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:30.028486 29549 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:30.028537 29549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:30.029302 29549 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4aea9b0a18459cb8cea0887b296a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38897 } }
I20260926 15:04:30.029415 29549 leader_election.cc:302] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4f4aea9b0a18459cb8cea0887b296a84; no voters:
I20260926 15:04:30.029587 29549 leader_election.cc:288] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:30.029659 29554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:30.029873 29549 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:30.029882 29554 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84 [term 1 LEADER]: Becoming Leader. State: Replica: 4f4aea9b0a18459cb8cea0887b296a84, State: Running, Role: LEADER
I20260926 15:04:30.030167 29554 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4aea9b0a18459cb8cea0887b296a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38897 } }
I20260926 15:04:30.030576 29556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4f4aea9b0a18459cb8cea0887b296a84. Latest consensus state: current_term: 1 leader_uuid: "4f4aea9b0a18459cb8cea0887b296a84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4aea9b0a18459cb8cea0887b296a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38897 } } }
I20260926 15:04:30.030584 29555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4f4aea9b0a18459cb8cea0887b296a84" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4f4aea9b0a18459cb8cea0887b296a84" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38897 } } }
I20260926 15:04:30.030730 29556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:30.031572 29560 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:30.030763 29555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:30.032125 29560 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 15:04:30 dist-test-slave-jnhs krb5kdc[29468](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435069, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 15:04:30.033748 29560 catalog_manager.cc:1347] Generated new cluster ID: c6126fff189143ccb654b03fa8a37b78
I20260926 15:04:30.033798 29560 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:30.037547 29552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:30.024503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:50507 (local address 127.0.0.1:38897)
0926 15:04:30.024722 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:30.024733 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:30.024772 (+ 39us) server_negotiation.cc:407] Connection header received
0926 15:04:30.025440 (+ 668us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:30.025452 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:30.025537 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:30.025699 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:30.026324 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:30.027948 (+ 1624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:30.029009 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:30.029680 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:30.029795 (+ 115us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:30.033438 (+ 3643us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:30.033442 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:30.033443 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:30.033456 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:30.035543 (+ 2087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:30.036182 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:30.036187 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:30.036187 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:30.036300 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:30.036565 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:30.036565 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:30.036566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:30.036910 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:30.037072 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:30.037224 (+ 152us) server_negotiation.cc:299] Negotiation successful
0926 15:04:30.037326 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":91,"threads_started":1}
I20260926 15:04:30.038826 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:04:30.049299 29560 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:30.049896 29560 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:30.054116 29560 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4f4aea9b0a18459cb8cea0887b296a84: Generated new TSK 0
I20260926 15:04:30.054414 29560 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:30.058758 29552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:30.040358 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:34463 (local address 127.0.0.1:38897)
0926 15:04:30.040394 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:30.040396 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:30.048946 (+ 8550us) server_negotiation.cc:407] Connection header received
0926 15:04:30.049026 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:30.049026 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:30.049038 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:30.049100 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:30.050737 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:30.051815 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:30.052924 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:30.053221 (+ 297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:30.053297 (+ 76us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:30.054942 (+ 1645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:30.054943 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:30.054943 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:30.054950 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:30.056666 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:30.057509 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:30.057510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:30.057510 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:30.057591 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:30.057952 (+ 361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:30.057954 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:30.057954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:30.058152 (+ 198us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:30.058255 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:30.058496 (+ 241us) server_negotiation.cc:299] Negotiation successful
0926 15:04:30.058598 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260926 15:04:30.060981 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 29481
W20260926 15:04:30.066434 29466 connection.cc:535] client connection to 127.0.0.1:38897 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38897): Connection reset by peer (error 104)
2026-09-26T15:04:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1360 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:30 dist-test-slave-jnhs krb5kdc[29580](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:30 dist-test-slave-jnhs krb5kdc[29580](info): set up 2 sockets
Sep 26 15:04:30 dist-test-slave-jnhs krb5kdc[29580](info): commencing operation
krb5kdc: starting...
W20260926 15:04:31.103386 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.023s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:31 dist-test-slave-jnhs krb5kdc[29580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:04:31.230477 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42955
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35931
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42955
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:31.316776 29593 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:31.317035 29593 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:31.317076 29593 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:31.318331 29593 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:31.318408 29593 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:31.318434 29593 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:31.318456 29593 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:31.318485 29593 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:31.320041 29593 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35931
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42955
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42955
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29593
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:31.320333 29593 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:31.320571 29593 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260926 15:04:31.325788 29593 server_base.cc:1034] running on GCE node
W20260926 15:04:31.325767 29598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:31.325779 29599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:31.325767 29601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:31.326208 29593 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:31.326538 29593 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:31.327708 29593 hybrid_clock.cc:648] HybridClock initialized: now 1790435071327649 us; error 73 us; skew 500 ppm
Sep 26 15:04:31 dist-test-slave-jnhs krb5kdc[29580](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435071, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:31.330544 29593 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:04:31.331146 29593 webserver.cc:460] Webserver started at http://127.0.0.1:41589/ using document root <none> and password file <none>
I20260926 15:04:31.331400 29593 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:31.331471 29593 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:31.331631 29593 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:31.332741 29593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "aed8663fa8134a4b99d35f9b3fa14e95"
format_stamp: "Formatted at 2026-09-26 15:04:31 on dist-test-slave-jnhs"
server_key: "53e092b4779174ff54bd6db07c2e725f"
server_key_iv: "a3076fec860f11556d10c02154542a10"
server_key_version: "encryptionkey@0"
I20260926 15:04:31.333115 29593 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "aed8663fa8134a4b99d35f9b3fa14e95"
format_stamp: "Formatted at 2026-09-26 15:04:31 on dist-test-slave-jnhs"
server_key: "53e092b4779174ff54bd6db07c2e725f"
server_key_iv: "a3076fec860f11556d10c02154542a10"
server_key_version: "encryptionkey@0"
I20260926 15:04:31.334522 29593 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:04:31.335354 29608 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:31.335641 29593 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:04:31.335752 29593 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "aed8663fa8134a4b99d35f9b3fa14e95"
format_stamp: "Formatted at 2026-09-26 15:04:31 on dist-test-slave-jnhs"
server_key: "53e092b4779174ff54bd6db07c2e725f"
server_key_iv: "a3076fec860f11556d10c02154542a10"
server_key_version: "encryptionkey@0"
I20260926 15:04:31.335834 29593 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:31.351464 29593 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:31.353637 29593 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:31.353822 29593 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:31.358048 29593 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42955
I20260926 15:04:31.358098 29660 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42955 every 8 connection(s)
I20260926 15:04:31.358476 29593 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:31.359534 29661 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:31.361779 29661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95: Bootstrap starting.
I20260926 15:04:31.362335 29661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:31.362560 29661 log.cc:824] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:31.363507 29661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95: No bootstrap required, opened a new log
I20260926 15:04:31.365203 29661 raft_consensus.cc:348] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed8663fa8134a4b99d35f9b3fa14e95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42955 } }
I20260926 15:04:31.365348 29661 raft_consensus.cc:374] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:31.365371 29661 raft_consensus.cc:729] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aed8663fa8134a4b99d35f9b3fa14e95, State: Initialized, Role: FOLLOWER
I20260926 15:04:31.365474 29661 consensus_queue.cc:260] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed8663fa8134a4b99d35f9b3fa14e95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42955 } }
I20260926 15:04:31.365528 29661 raft_consensus.cc:388] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:31.365548 29661 raft_consensus.cc:482] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:31.365638 29661 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:31.366163 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 29593
I20260926 15:04:31.366189 29661 raft_consensus.cc:504] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed8663fa8134a4b99d35f9b3fa14e95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42955 } }
I20260926 15:04:31.366266 29661 leader_election.cc:302] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: aed8663fa8134a4b99d35f9b3fa14e95; no voters:
I20260926 15:04:31.366308 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:31.366496 29661 leader_election.cc:288] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:31.366521 29664 raft_consensus.cc:2781] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:31.366530 8600 external_mini_cluster.cc:1428] Setting key 79cab89e5dbb5ed57e97479a56045875
I20260926 15:04:31.366680 29664 raft_consensus.cc:686] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95 [term 1 LEADER]: Becoming Leader. State: Replica: aed8663fa8134a4b99d35f9b3fa14e95, State: Running, Role: LEADER
I20260926 15:04:31.366815 29661 sys_catalog.cc:564] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:31.366837 29664 consensus_queue.cc:237] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed8663fa8134a4b99d35f9b3fa14e95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42955 } }
I20260926 15:04:31.367282 29665 sys_catalog.cc:455] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "aed8663fa8134a4b99d35f9b3fa14e95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed8663fa8134a4b99d35f9b3fa14e95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42955 } } }
I20260926 15:04:31.367444 29666 sys_catalog.cc:455] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95 [sys.catalog]: SysCatalogTable state changed. Reason: New leader aed8663fa8134a4b99d35f9b3fa14e95. Latest consensus state: current_term: 1 leader_uuid: "aed8663fa8134a4b99d35f9b3fa14e95" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "aed8663fa8134a4b99d35f9b3fa14e95" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42955 } } }
I20260926 15:04:31.367525 29665 sys_catalog.cc:458] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:31.367784 29666 sys_catalog.cc:458] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:31.367965 29671 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:31.368690 29671 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:31.370587 29671 catalog_manager.cc:1347] Generated new cluster ID: c7b0c859bb6344e3a4804f2a30847793
I20260926 15:04:31.370675 29671 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:31.373103 29669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:31.367261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:41311 (local address 127.0.0.1:42955)
0926 15:04:31.367477 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:31.367485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:31.367509 (+ 24us) server_negotiation.cc:407] Connection header received
0926 15:04:31.367655 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:31.367663 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:31.367734 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:31.368147 (+ 413us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:31.368277 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:31.369653 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:31.370839 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:31.371485 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:31.371618 (+ 133us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:31.372382 (+ 764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:31.372386 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:31.372386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:31.372612 (+ 226us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:31.372655 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:31.372672 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:31.372766 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:31.372849 (+ 83us) server_negotiation.cc:299] Negotiation successful
0926 15:04:31.372879 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":84,"threads_started":1}
I20260926 15:04:31.374202 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:04:31.379432 29669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:31.375677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:58663 (local address 127.0.0.1:42955)
0926 15:04:31.375718 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:31.375720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:31.375767 (+ 47us) server_negotiation.cc:407] Connection header received
0926 15:04:31.375830 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:31.375831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:31.375841 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:31.375894 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:31.376068 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:31.376834 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:31.377902 (+ 1068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:31.378213 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:31.378276 (+ 63us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:31.379051 (+ 775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:31.379053 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:31.379053 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:31.379058 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:31.379073 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:31.379077 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:31.379135 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:31.379338 (+ 203us) server_negotiation.cc:299] Negotiation successful
0926 15:04:31.379352 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 15:04:31.387107 29671 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:31.387691 29671 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:31.395712 29671 catalog_manager.cc:5868] T 00000000000000000000000000000000 P aed8663fa8134a4b99d35f9b3fa14e95: Generated new TSK 0
I20260926 15:04:31.395934 29671 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:31.399421 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 29593
W20260926 15:04:31.404454 29578 connection.cc:535] client connection to 127.0.0.1:42955 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42955): Connection reset by peer (error 104)
2026-09-26T15:04:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1337 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:31 dist-test-slave-jnhs krb5kdc[29692](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:31 dist-test-slave-jnhs krb5kdc[29692](info): set up 2 sockets
Sep 26 15:04:31 dist-test-slave-jnhs krb5kdc[29692](info): commencing operation
krb5kdc: starting...
W20260926 15:04:32.450922 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:32 dist-test-slave-jnhs krb5kdc[29692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435072, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:04:32.577646 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41365
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39921
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:32.658026 29705 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:32.658250 29705 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:32.658275 29705 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:32.659404 29705 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:32.659447 29705 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:32.659462 29705 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:32.659475 29705 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:32.659487 29705 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:32.660914 29705 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39921
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41365
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29705
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:32.661114 29705 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:32.661283 29705 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:32.666465 29711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:32.666460 29710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:32.666474 29713 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:32.666767 29705 server_base.cc:1034] running on GCE node
I20260926 15:04:32.667034 29705 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:32.667368 29705 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:32.668576 29705 hybrid_clock.cc:648] HybridClock initialized: now 1790435072668542 us; error 53 us; skew 500 ppm
Sep 26 15:04:32 dist-test-slave-jnhs krb5kdc[29692](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435072, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:32.671397 29705 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:04:32.671865 29705 webserver.cc:460] Webserver started at http://127.0.0.1:46649/ using document root <none> and password file <none>
I20260926 15:04:32.672127 29705 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:32.672226 29705 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:32.672369 29705 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:32.673447 29705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "788d35b909ab48978550c1bc48701a16"
format_stamp: "Formatted at 2026-09-26 15:04:32 on dist-test-slave-jnhs"
server_key: "a5dda6490261ecc27c3c10f69c4d42c7"
server_key_iv: "3fb8eb2112ce44338d2bc4ae854a2ecc"
server_key_version: "encryptionkey@0"
I20260926 15:04:32.673811 29705 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "788d35b909ab48978550c1bc48701a16"
format_stamp: "Formatted at 2026-09-26 15:04:32 on dist-test-slave-jnhs"
server_key: "a5dda6490261ecc27c3c10f69c4d42c7"
server_key_iv: "3fb8eb2112ce44338d2bc4ae854a2ecc"
server_key_version: "encryptionkey@0"
I20260926 15:04:32.675318 29705 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:04:32.676165 29720 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:32.676352 29705 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:32.676461 29705 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "788d35b909ab48978550c1bc48701a16"
format_stamp: "Formatted at 2026-09-26 15:04:32 on dist-test-slave-jnhs"
server_key: "a5dda6490261ecc27c3c10f69c4d42c7"
server_key_iv: "3fb8eb2112ce44338d2bc4ae854a2ecc"
server_key_version: "encryptionkey@0"
I20260926 15:04:32.676544 29705 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:32.680049 29705 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:32.682484 29705 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:32.682718 29705 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:32.686427 29705 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41365
I20260926 15:04:32.686501 29772 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41365 every 8 connection(s)
I20260926 15:04:32.686864 29705 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:32.687901 29773 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:32.690300 29773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16: Bootstrap starting.
I20260926 15:04:32.690984 29773 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:32.691243 29773 log.cc:824] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:32.691977 29773 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16: No bootstrap required, opened a new log
I20260926 15:04:32.692725 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 29705
I20260926 15:04:32.692852 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:32.693073 8600 external_mini_cluster.cc:1428] Setting key 8ff78c63284bc6e856163adcb66768ed
I20260926 15:04:32.694377 29773 raft_consensus.cc:348] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788d35b909ab48978550c1bc48701a16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41365 } }
I20260926 15:04:32.694566 29773 raft_consensus.cc:374] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:32.694610 29773 raft_consensus.cc:729] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 788d35b909ab48978550c1bc48701a16, State: Initialized, Role: FOLLOWER
I20260926 15:04:32.694847 29773 consensus_queue.cc:260] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788d35b909ab48978550c1bc48701a16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41365 } }
I20260926 15:04:32.694958 29773 raft_consensus.cc:388] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:32.694986 29773 raft_consensus.cc:482] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:32.695065 29773 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:32.695912 29777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:32.693720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:58147 (local address 127.0.0.1:41365)
0926 15:04:32.693904 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:32.693912 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:32.693941 (+ 29us) server_negotiation.cc:407] Connection header received
0926 15:04:32.694373 (+ 432us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:32.694381 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:32.694414 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:32.694604 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:32.695395 (+ 791us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:32.695398 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:32.695399 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:32.695402 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:32.695442 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:32.695453 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:32.695517 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:32.695621 (+ 104us) server_negotiation.cc:299] Negotiation successful
0926 15:04:32.695678 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":85,"threads_started":1}
I20260926 15:04:32.696383 29773 raft_consensus.cc:504] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788d35b909ab48978550c1bc48701a16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41365 } }
I20260926 15:04:32.696513 29773 leader_election.cc:302] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 788d35b909ab48978550c1bc48701a16; no voters:
I20260926 15:04:32.696743 29773 leader_election.cc:288] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:32.696863 29778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:32.696933 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:04:32.696992 29778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16 [term 1 LEADER]: Becoming Leader. State: Replica: 788d35b909ab48978550c1bc48701a16, State: Running, Role: LEADER
I20260926 15:04:32.697060 29773 sys_catalog.cc:564] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:32.697151 29778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788d35b909ab48978550c1bc48701a16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41365 } }
I20260926 15:04:32.697548 29778 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 788d35b909ab48978550c1bc48701a16. Latest consensus state: current_term: 1 leader_uuid: "788d35b909ab48978550c1bc48701a16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788d35b909ab48978550c1bc48701a16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41365 } } }
I20260926 15:04:32.697561 29780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "788d35b909ab48978550c1bc48701a16" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "788d35b909ab48978550c1bc48701a16" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41365 } } }
I20260926 15:04:32.697733 29780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:32.697950 29778 sys_catalog.cc:458] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16 [sys.catalog]: This master's current role is: LEADER
W20260926 15:04:32.699124 29798 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:04:32.699182 29798 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:04:32.699226 29787 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:32.699368 29787 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:32.701002 29787 catalog_manager.cc:1347] Generated new cluster ID: d90bb5b4642742c7842ac95fe8f34d43
I20260926 15:04:32.701081 29787 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:32.706984 29777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:32.699051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:56411 (local address 127.0.0.1:41365)
0926 15:04:32.699093 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:32.699096 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:32.705847 (+ 6751us) server_negotiation.cc:407] Connection header received
0926 15:04:32.705923 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:32.705924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:32.705931 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:32.706008 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:32.706730 (+ 722us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:32.706732 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:32.706732 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:32.706736 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:32.706754 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:32.706758 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:32.706815 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:32.706893 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 15:04:32.706908 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 15:04:32.724512 29787 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:32.724848 29787 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:32.728596 29787 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 788d35b909ab48978550c1bc48701a16: Generated new TSK 0
I20260926 15:04:32.728853 29787 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:32.734217 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 29705
2026-09-26T15:04:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1334 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:32 dist-test-slave-jnhs krb5kdc[29804](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:32 dist-test-slave-jnhs krb5kdc[29804](info): set up 2 sockets
Sep 26 15:04:32 dist-test-slave-jnhs krb5kdc[29804](info): commencing operation
krb5kdc: starting...
W20260926 15:04:33.819422 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.067s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:33 dist-test-slave-jnhs krb5kdc[29804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:04:33.946273 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42279
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:34.028640 29817 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:34.028909 29817 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:34.028949 29817 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:34.030154 29817 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:34.030233 29817 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:34.030258 29817 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:34.030282 29817 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:34.030311 29817 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:34.031930 29817 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42279
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29817
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:34.032248 29817 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:34.032495 29817 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:34.037887 29825 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:34.037890 29822 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:34.037894 29823 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:34.038324 29817 server_base.cc:1034] running on GCE node
I20260926 15:04:34.038766 29817 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:34.039062 29817 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:34.040253 29817 hybrid_clock.cc:648] HybridClock initialized: now 1790435074040240 us; error 31 us; skew 500 ppm
Sep 26 15:04:34 dist-test-slave-jnhs krb5kdc[29804](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435074, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:34.052613 29817 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:04:34.053264 29817 webserver.cc:460] Webserver started at http://127.0.0.1:45095/ using document root <none> and password file <none>
I20260926 15:04:34.053550 29817 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:34.053627 29817 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:34.053810 29817 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:34.055111 29817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "0e14bb48a29e4ff98d0617f15c3c8f03"
format_stamp: "Formatted at 2026-09-26 15:04:34 on dist-test-slave-jnhs"
server_key: "17226756e4f61a7e7baff1ecdf0d78c0"
server_key_iv: "8fb4e954904b96b6f48ee4a67b3a231c"
server_key_version: "encryptionkey@0"
I20260926 15:04:34.055500 29817 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "0e14bb48a29e4ff98d0617f15c3c8f03"
format_stamp: "Formatted at 2026-09-26 15:04:34 on dist-test-slave-jnhs"
server_key: "17226756e4f61a7e7baff1ecdf0d78c0"
server_key_iv: "8fb4e954904b96b6f48ee4a67b3a231c"
server_key_version: "encryptionkey@0"
I20260926 15:04:34.057067 29817 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:04:34.058017 29832 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:34.058353 29817 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:34.058476 29817 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "0e14bb48a29e4ff98d0617f15c3c8f03"
format_stamp: "Formatted at 2026-09-26 15:04:34 on dist-test-slave-jnhs"
server_key: "17226756e4f61a7e7baff1ecdf0d78c0"
server_key_iv: "8fb4e954904b96b6f48ee4a67b3a231c"
server_key_version: "encryptionkey@0"
I20260926 15:04:34.058558 29817 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:34.067178 29817 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:34.069703 29817 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:34.069893 29817 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:34.074380 29817 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42279
I20260926 15:04:34.074420 29884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42279 every 8 connection(s)
I20260926 15:04:34.074888 29817 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:34.075932 29885 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:34.078460 29885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03: Bootstrap starting.
I20260926 15:04:34.079159 29885 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:34.079474 29885 log.cc:824] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:34.080328 29885 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03: No bootstrap required, opened a new log
I20260926 15:04:34.081825 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 29817
I20260926 15:04:34.081962 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:34.082182 8600 external_mini_cluster.cc:1428] Setting key 3d084d7ccedc30545185dbc6f52752ea
I20260926 15:04:34.082563 29885 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e14bb48a29e4ff98d0617f15c3c8f03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42279 } }
I20260926 15:04:34.082831 29885 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:34.082890 29885 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e14bb48a29e4ff98d0617f15c3c8f03, State: Initialized, Role: FOLLOWER
I20260926 15:04:34.083060 29885 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e14bb48a29e4ff98d0617f15c3c8f03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42279 } }
I20260926 15:04:34.083163 29885 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:34.083212 29885 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:34.083268 29885 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:34.084591 29885 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e14bb48a29e4ff98d0617f15c3c8f03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42279 } }
I20260926 15:04:34.084728 29885 leader_election.cc:302] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e14bb48a29e4ff98d0617f15c3c8f03; no voters:
I20260926 15:04:34.084928 29885 leader_election.cc:288] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:34.084976 29890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:34.085135 29890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03 [term 1 LEADER]: Becoming Leader. State: Replica: 0e14bb48a29e4ff98d0617f15c3c8f03, State: Running, Role: LEADER
I20260926 15:04:34.085238 29885 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:34.085247 29890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e14bb48a29e4ff98d0617f15c3c8f03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42279 } }
I20260926 15:04:34.085604 29890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e14bb48a29e4ff98d0617f15c3c8f03. Latest consensus state: current_term: 1 leader_uuid: "0e14bb48a29e4ff98d0617f15c3c8f03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e14bb48a29e4ff98d0617f15c3c8f03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42279 } } }
I20260926 15:04:34.085664 29890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:34.085630 29892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e14bb48a29e4ff98d0617f15c3c8f03" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e14bb48a29e4ff98d0617f15c3c8f03" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42279 } } }
I20260926 15:04:34.085861 29895 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:34.085868 29892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:34.086390 29895 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 26 15:04:34 dist-test-slave-jnhs krb5kdc[29804](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260926 15:04:34.092120 29895 catalog_manager.cc:1347] Generated new cluster ID: 7ec373b808ce476aa22e96cb05d63a8e
I20260926 15:04:34.092196 29895 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:34.095024 29889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:34.082784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:55803 (local address 127.0.0.1:42279)
0926 15:04:34.083070 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:34.083079 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:34.083115 (+ 36us) server_negotiation.cc:407] Connection header received
0926 15:04:34.083244 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:34.083251 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:34.083313 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:34.083459 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:34.084393 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:34.085777 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:34.086846 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:34.087482 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:34.087636 (+ 154us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:34.090543 (+ 2907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:34.090548 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:34.090549 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:34.090562 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:34.092952 (+ 2390us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:34.093422 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:34.093426 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:34.093427 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:34.093579 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:34.093818 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:34.093819 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:34.093819 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:34.094172 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:34.094355 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:34.094451 (+ 96us) server_negotiation.cc:299] Negotiation successful
0926 15:04:34.094581 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":221,"thread_start_us":115,"threads_started":1}
I20260926 15:04:34.096269 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:04:34.101388 29895 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:34.101999 29895 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:34.105898 29895 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0e14bb48a29e4ff98d0617f15c3c8f03: Generated new TSK 0
I20260926 15:04:34.106112 29895 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:34.106731 29889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:34.097748 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:58049 (local address 127.0.0.1:42279)
0926 15:04:34.097783 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:34.097785 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:34.098810 (+ 1025us) server_negotiation.cc:407] Connection header received
0926 15:04:34.098844 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:34.098845 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:34.098857 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:34.098927 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:34.099563 (+ 636us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:34.100490 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:34.101389 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:34.101788 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:34.101871 (+ 83us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:34.103358 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:34.103360 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:34.103360 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:34.103368 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:34.105149 (+ 1781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:34.105786 (+ 637us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:34.105787 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:34.105788 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:34.105852 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:34.106147 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:34.106149 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:34.106149 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:34.106301 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:34.106415 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:34.106537 (+ 122us) server_negotiation.cc:299] Negotiation successful
0926 15:04:34.106610 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260926 15:04:34.108975 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 29817
W20260926 15:04:34.112929 29802 connection.cc:535] client connection to 127.0.0.1:42279 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42279): Connection reset by peer (error 104)
2026-09-26T15:04:34Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1374 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:34 dist-test-slave-jnhs krb5kdc[29916](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:34 dist-test-slave-jnhs krb5kdc[29916](info): set up 2 sockets
Sep 26 15:04:34 dist-test-slave-jnhs krb5kdc[29916](info): commencing operation
krb5kdc: starting...
W20260926 15:04:35.163552 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.037s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:35 dist-test-slave-jnhs krb5kdc[29916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435075, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:04:35.295886 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35765
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34045
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35765
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:35.381237 29929 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:35.381453 29929 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:35.381477 29929 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:35.382761 29929 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:35.382814 29929 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:35.382829 29929 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:35.382843 29929 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:35.382854 29929 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:35.384467 29929 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34045
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35765
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35765
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29929
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:35.384770 29929 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:35.385035 29929 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:35.390610 29937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:35.390609 29935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:35.390779 29929 server_base.cc:1034] running on GCE node
W20260926 15:04:35.390604 29934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:35.391197 29929 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:35.391489 29929 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:35.392639 29929 hybrid_clock.cc:648] HybridClock initialized: now 1790435075392622 us; error 27 us; skew 500 ppm
Sep 26 15:04:35 dist-test-slave-jnhs krb5kdc[29916](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435075, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:35.395292 29929 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:04:35.395802 29929 webserver.cc:460] Webserver started at http://127.0.0.1:43177/ using document root <none> and password file <none>
I20260926 15:04:35.396011 29929 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:35.396077 29929 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:35.396170 29929 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:35.397212 29929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "e11f30ad39e049fea77f85febb271854"
format_stamp: "Formatted at 2026-09-26 15:04:35 on dist-test-slave-jnhs"
server_key: "9778202b704b7f186e4d4a22d5574fe4"
server_key_iv: "4960fd187e65743ae6f0a676906e280f"
server_key_version: "encryptionkey@0"
I20260926 15:04:35.397523 29929 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "e11f30ad39e049fea77f85febb271854"
format_stamp: "Formatted at 2026-09-26 15:04:35 on dist-test-slave-jnhs"
server_key: "9778202b704b7f186e4d4a22d5574fe4"
server_key_iv: "4960fd187e65743ae6f0a676906e280f"
server_key_version: "encryptionkey@0"
I20260926 15:04:35.399013 29929 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 15:04:35.399950 29944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:35.400198 29929 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:35.400276 29929 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "e11f30ad39e049fea77f85febb271854"
format_stamp: "Formatted at 2026-09-26 15:04:35 on dist-test-slave-jnhs"
server_key: "9778202b704b7f186e4d4a22d5574fe4"
server_key_iv: "4960fd187e65743ae6f0a676906e280f"
server_key_version: "encryptionkey@0"
I20260926 15:04:35.400379 29929 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:35.428983 29929 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:35.431384 29929 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:35.431552 29929 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:35.435860 29929 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35765
I20260926 15:04:35.435947 29996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35765 every 8 connection(s)
I20260926 15:04:35.436305 29929 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:35.437237 29997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:35.439677 29997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854: Bootstrap starting.
I20260926 15:04:35.440240 29997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:35.440488 29997 log.cc:824] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:35.441246 29997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854: No bootstrap required, opened a new log
I20260926 15:04:35.441597 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 29929
I20260926 15:04:35.441712 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:35.441929 8600 external_mini_cluster.cc:1428] Setting key bd520a015a61553244676008ff7d65ce
I20260926 15:04:35.443818 29997 raft_consensus.cc:348] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11f30ad39e049fea77f85febb271854" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35765 } }
I20260926 15:04:35.444005 29997 raft_consensus.cc:374] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:35.444056 29997 raft_consensus.cc:729] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e11f30ad39e049fea77f85febb271854, State: Initialized, Role: FOLLOWER
I20260926 15:04:35.444201 29997 consensus_queue.cc:260] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11f30ad39e049fea77f85febb271854" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35765 } }
I20260926 15:04:35.444307 29997 raft_consensus.cc:388] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:35.444351 29997 raft_consensus.cc:482] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:35.444401 29997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:35.445000 29997 raft_consensus.cc:504] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11f30ad39e049fea77f85febb271854" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35765 } }
I20260926 15:04:35.445187 29997 leader_election.cc:302] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e11f30ad39e049fea77f85febb271854; no voters:
I20260926 15:04:35.445379 29997 leader_election.cc:288] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:35.445426 30002 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:35.445586 30002 raft_consensus.cc:686] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854 [term 1 LEADER]: Becoming Leader. State: Replica: e11f30ad39e049fea77f85febb271854, State: Running, Role: LEADER
I20260926 15:04:35.445670 29997 sys_catalog.cc:564] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:35.445711 30002 consensus_queue.cc:237] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11f30ad39e049fea77f85febb271854" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35765 } }
I20260926 15:04:35.446171 30003 sys_catalog.cc:455] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e11f30ad39e049fea77f85febb271854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11f30ad39e049fea77f85febb271854" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35765 } } }
I20260926 15:04:35.446265 30003 sys_catalog.cc:458] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:35.446463 30004 sys_catalog.cc:455] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e11f30ad39e049fea77f85febb271854. Latest consensus state: current_term: 1 leader_uuid: "e11f30ad39e049fea77f85febb271854" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e11f30ad39e049fea77f85febb271854" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35765 } } }
I20260926 15:04:35.446784 30004 sys_catalog.cc:458] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:35.446872 30009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:35.447372 30009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:35.448802 30001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:35.442533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:33425 (local address 127.0.0.1:35765)
0926 15:04:35.442744 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:35.442753 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:35.442780 (+ 27us) server_negotiation.cc:407] Connection header received
0926 15:04:35.443251 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:35.443259 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:35.443314 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:35.443436 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:35.443791 (+ 355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:35.445177 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:35.446335 (+ 1158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:35.447075 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:35.447169 (+ 94us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:35.447981 (+ 812us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:35.447986 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:35.447986 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:35.448208 (+ 222us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:35.448251 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:35.448266 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:35.448389 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:35.448473 (+ 84us) server_negotiation.cc:299] Negotiation successful
0926 15:04:35.448503 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":92,"threads_started":1}
I20260926 15:04:35.449235 30009 catalog_manager.cc:1347] Generated new cluster ID: 71b2eee52d2447d0b7c7b048a26881b0
I20260926 15:04:35.449285 30009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:35.449793 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:04:35.455255 30001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:35.451341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:34599 (local address 127.0.0.1:35765)
0926 15:04:35.451380 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:35.451381 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:35.451494 (+ 113us) server_negotiation.cc:407] Connection header received
0926 15:04:35.451588 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:35.451589 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:35.451599 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:35.451628 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:35.451867 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:35.452591 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:35.453690 (+ 1099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:35.454027 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:35.454071 (+ 44us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:35.454946 (+ 875us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:35.454947 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:35.454948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:35.454953 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:35.454968 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:35.454972 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:35.455006 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:35.455147 (+ 141us) server_negotiation.cc:299] Negotiation successful
0926 15:04:35.455160 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 15:04:35.455871 30009 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:35.456478 30009 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:35.458689 30009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e11f30ad39e049fea77f85febb271854: Generated new TSK 0
I20260926 15:04:35.458894 30009 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:35.462543 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 29929
W20260926 15:04:35.467562 29914 connection.cc:535] client connection to 127.0.0.1:35765 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:35765): Connection reset by peer (error 104)
2026-09-26T15:04:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1355 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:35 dist-test-slave-jnhs krb5kdc[30028](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:35 dist-test-slave-jnhs krb5kdc[30028](info): set up 2 sockets
Sep 26 15:04:35 dist-test-slave-jnhs krb5kdc[30028](info): commencing operation
krb5kdc: starting...
W20260926 15:04:36.550940 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.069s user 0.003s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:36 dist-test-slave-jnhs krb5kdc[30028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435076, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260926 15:04:36.688334 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35219
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40409
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:36.774662 30041 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:36.774932 30041 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:36.774976 30041 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:36.776189 30041 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:36.776280 30041 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:36.776307 30041 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:36.776341 30041 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:36.776371 30041 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:36.777885 30041 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40409
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35219
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30041
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:36.778184 30041 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:36.778430 30041 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:36.783808 30049 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:36.783921 30041 server_base.cc:1034] running on GCE node
W20260926 15:04:36.783856 30047 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:36.783836 30046 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:36.784301 30041 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:36.784576 30041 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:36.785730 30041 hybrid_clock.cc:648] HybridClock initialized: now 1790435076785693 us; error 59 us; skew 500 ppm
Sep 26 15:04:36 dist-test-slave-jnhs krb5kdc[30028](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435076, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:36.788533 30041 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260926 15:04:36.789057 30041 webserver.cc:460] Webserver started at http://127.0.0.1:39075/ using document root <none> and password file <none>
I20260926 15:04:36.789299 30041 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:36.789372 30041 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:36.789522 30041 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:36.790735 30041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "1a1ea9f1d15147788051f4b60ce2f584"
format_stamp: "Formatted at 2026-09-26 15:04:36 on dist-test-slave-jnhs"
server_key: "ee9dc48d6dcdfbf76f856b3cd22e68f0"
server_key_iv: "2ad65835dd7f021f535c5e7813e66973"
server_key_version: "encryptionkey@0"
I20260926 15:04:36.791111 30041 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "1a1ea9f1d15147788051f4b60ce2f584"
format_stamp: "Formatted at 2026-09-26 15:04:36 on dist-test-slave-jnhs"
server_key: "ee9dc48d6dcdfbf76f856b3cd22e68f0"
server_key_iv: "2ad65835dd7f021f535c5e7813e66973"
server_key_version: "encryptionkey@0"
I20260926 15:04:36.792507 30041 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 15:04:36.793375 30056 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:36.793553 30041 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:04:36.793637 30041 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "1a1ea9f1d15147788051f4b60ce2f584"
format_stamp: "Formatted at 2026-09-26 15:04:36 on dist-test-slave-jnhs"
server_key: "ee9dc48d6dcdfbf76f856b3cd22e68f0"
server_key_iv: "2ad65835dd7f021f535c5e7813e66973"
server_key_version: "encryptionkey@0"
I20260926 15:04:36.793692 30041 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:36.797082 30041 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:36.799286 30041 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:36.799399 30041 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:36.803649 30041 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35219
I20260926 15:04:36.803681 30108 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35219 every 8 connection(s)
I20260926 15:04:36.804023 30041 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:36.804719 30109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:36.807077 30109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584: Bootstrap starting.
I20260926 15:04:36.807679 30109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:36.807976 30109 log.cc:824] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:36.808737 30109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584: No bootstrap required, opened a new log
I20260926 15:04:36.810352 30109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1ea9f1d15147788051f4b60ce2f584" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35219 } }
I20260926 15:04:36.810526 30109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:36.810580 30109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a1ea9f1d15147788051f4b60ce2f584, State: Initialized, Role: FOLLOWER
I20260926 15:04:36.810794 30109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1ea9f1d15147788051f4b60ce2f584" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35219 } }
I20260926 15:04:36.810892 30109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:36.810938 30109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:36.810990 30109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:36.811550 30109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1ea9f1d15147788051f4b60ce2f584" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35219 } }
I20260926 15:04:36.811686 30109 leader_election.cc:302] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a1ea9f1d15147788051f4b60ce2f584; no voters:
I20260926 15:04:36.811873 30109 leader_election.cc:288] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:36.811930 30112 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:36.812147 30109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:36.812175 30112 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584 [term 1 LEADER]: Becoming Leader. State: Replica: 1a1ea9f1d15147788051f4b60ce2f584, State: Running, Role: LEADER
I20260926 15:04:36.812287 30112 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1ea9f1d15147788051f4b60ce2f584" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35219 } }
I20260926 15:04:36.813028 30114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a1ea9f1d15147788051f4b60ce2f584. Latest consensus state: current_term: 1 leader_uuid: "1a1ea9f1d15147788051f4b60ce2f584" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1ea9f1d15147788051f4b60ce2f584" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35219 } } }
I20260926 15:04:36.813113 30114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:36.813185 30113 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a1ea9f1d15147788051f4b60ce2f584" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a1ea9f1d15147788051f4b60ce2f584" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35219 } } }
I20260926 15:04:36.813269 30113 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:36.813692 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 30041
I20260926 15:04:36.813817 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:36.813628 30121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:36.814018 8600 external_mini_cluster.cc:1428] Setting key c4b7eea747e7d1dd45af4116f80442da
I20260926 15:04:36.814428 30121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:36.816445 30121 catalog_manager.cc:1347] Generated new cluster ID: ea2c189877ca42f785c561e5caf983ce
I20260926 15:04:36.816490 30121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:36.816643 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:36.814574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:37881 (local address 127.0.0.1:35219)
0926 15:04:36.814768 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:36.814778 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:36.814816 (+ 38us) server_negotiation.cc:407] Connection header received
0926 15:04:36.814940 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:36.814947 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:36.814976 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:36.815125 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:36.816198 (+ 1073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:36.816202 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:36.816202 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:36.816207 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:36.816247 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:36.816262 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:36.816334 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:36.816420 (+ 86us) server_negotiation.cc:299] Negotiation successful
0926 15:04:36.816463 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":130,"thread_start_us":103,"threads_started":1}
I20260926 15:04:36.817623 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:04:36.823896 30131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:36.819210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.30:49327 (local address 127.0.0.1:35219)
0926 15:04:36.819244 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:36.819246 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:36.822887 (+ 3641us) server_negotiation.cc:407] Connection header received
0926 15:04:36.822953 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:36.822954 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:36.822972 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:36.823031 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:36.823654 (+ 623us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:36.823656 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:36.823656 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:36.823660 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:36.823675 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:36.823680 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:36.823732 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:36.823807 (+ 75us) server_negotiation.cc:299] Negotiation successful
0926 15:04:36.823820 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 15:04:36.825992 30121 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:36.826169 30121 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:36.830143 30121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a1ea9f1d15147788051f4b60ce2f584: Generated new TSK 0
I20260926 15:04:36.830323 30121 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:36.832664 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 30041
2026-09-26T15:04:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:36 dist-test-slave-jnhs krb5kdc[30140](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:36 dist-test-slave-jnhs krb5kdc[30140](info): set up 2 sockets
Sep 26 15:04:36 dist-test-slave-jnhs krb5kdc[30140](info): commencing operation
krb5kdc: starting...
W20260926 15:04:37.894914 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.044s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:37 dist-test-slave-jnhs krb5kdc[30140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:04:38.027029 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:38797
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:42181
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:38797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:38.111807 30153 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:38.112087 30153 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:38.112138 30153 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:38.113337 30153 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:38.113425 30153 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:38.113452 30153 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:38.113481 30153 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:38.113500 30153 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:38.115056 30153 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:42181
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:38797
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:38797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30153
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:38.115336 30153 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:38.115576 30153 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:38.120692 30161 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:38.120664 30159 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:38.120695 30158 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:38.120910 30153 server_base.cc:1034] running on GCE node
I20260926 15:04:38.121244 30153 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:38.121520 30153 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:38.122727 30153 hybrid_clock.cc:648] HybridClock initialized: now 1790435078122664 us; error 58 us; skew 500 ppm
Sep 26 15:04:38 dist-test-slave-jnhs krb5kdc[30140](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435078, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:38.125355 30153 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:04:38.125806 30153 webserver.cc:460] Webserver started at http://127.8.102.62:34149/ using document root <none> and password file <none>
I20260926 15:04:38.126055 30153 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:38.126106 30153 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:38.126243 30153 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:38.127424 30153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "b11ef3e1a1ef4ebcbb0808942f0c6424"
format_stamp: "Formatted at 2026-09-26 15:04:38 on dist-test-slave-jnhs"
server_key: "dc783059e55c0853c3b60f6299ad2b26"
server_key_iv: "127ac098355e16d85d5c1548cf622137"
server_key_version: "encryptionkey@0"
I20260926 15:04:38.127807 30153 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "b11ef3e1a1ef4ebcbb0808942f0c6424"
format_stamp: "Formatted at 2026-09-26 15:04:38 on dist-test-slave-jnhs"
server_key: "dc783059e55c0853c3b60f6299ad2b26"
server_key_iv: "127ac098355e16d85d5c1548cf622137"
server_key_version: "encryptionkey@0"
I20260926 15:04:38.129242 30153 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:04:38.130151 30168 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:38.130362 30153 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:38.130486 30153 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "b11ef3e1a1ef4ebcbb0808942f0c6424"
format_stamp: "Formatted at 2026-09-26 15:04:38 on dist-test-slave-jnhs"
server_key: "dc783059e55c0853c3b60f6299ad2b26"
server_key_iv: "127ac098355e16d85d5c1548cf622137"
server_key_version: "encryptionkey@0"
I20260926 15:04:38.130569 30153 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:38.160022 30153 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:38.162056 30153 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:38.162176 30153 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:38.166954 30153 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:38797
I20260926 15:04:38.167085 30220 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:38797 every 8 connection(s)
I20260926 15:04:38.167428 30153 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:38.168443 30221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:38.170867 30221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424: Bootstrap starting.
I20260926 15:04:38.171512 30221 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:38.171836 30221 log.cc:824] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:38.172708 30221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424: No bootstrap required, opened a new log
I20260926 15:04:38.172807 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 30153
I20260926 15:04:38.172968 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:38.173170 8600 external_mini_cluster.cc:1428] Setting key f6521a73cf762279e99c2548b387010c
I20260926 15:04:38.174909 30221 raft_consensus.cc:348] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11ef3e1a1ef4ebcbb0808942f0c6424" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38797 } }
I20260926 15:04:38.175089 30221 raft_consensus.cc:374] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:38.175156 30221 raft_consensus.cc:729] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b11ef3e1a1ef4ebcbb0808942f0c6424, State: Initialized, Role: FOLLOWER
I20260926 15:04:38.175325 30221 consensus_queue.cc:260] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11ef3e1a1ef4ebcbb0808942f0c6424" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38797 } }
I20260926 15:04:38.175424 30221 raft_consensus.cc:388] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:38.175472 30221 raft_consensus.cc:482] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:38.175525 30221 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:38.176486 30221 raft_consensus.cc:504] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11ef3e1a1ef4ebcbb0808942f0c6424" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38797 } }
I20260926 15:04:38.176618 30221 leader_election.cc:302] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b11ef3e1a1ef4ebcbb0808942f0c6424; no voters:
I20260926 15:04:38.176834 30221 leader_election.cc:288] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:38.176872 30226 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:38.176985 30226 raft_consensus.cc:686] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424 [term 1 LEADER]: Becoming Leader. State: Replica: b11ef3e1a1ef4ebcbb0808942f0c6424, State: Running, Role: LEADER
I20260926 15:04:38.177129 30221 sys_catalog.cc:564] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:38.177129 30226 consensus_queue.cc:237] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11ef3e1a1ef4ebcbb0808942f0c6424" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38797 } }
I20260926 15:04:38.177650 30227 sys_catalog.cc:455] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b11ef3e1a1ef4ebcbb0808942f0c6424" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11ef3e1a1ef4ebcbb0808942f0c6424" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38797 } } }
I20260926 15:04:38.177909 30227 sys_catalog.cc:458] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:38.177824 30228 sys_catalog.cc:455] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b11ef3e1a1ef4ebcbb0808942f0c6424. Latest consensus state: current_term: 1 leader_uuid: "b11ef3e1a1ef4ebcbb0808942f0c6424" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b11ef3e1a1ef4ebcbb0808942f0c6424" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38797 } } }
I20260926 15:04:38.178339 30228 sys_catalog.cc:458] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:38.178562 30241 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:38.179306 30241 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:38.181135 30241 catalog_manager.cc:1347] Generated new cluster ID: d9079f96b4644ca38e7c2483f46b9016
I20260926 15:04:38.181190 30241 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 26 15:04:38 dist-test-slave-jnhs krb5kdc[30140](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435077, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:04:38.186319 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:38.173777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42211 (local address 127.8.102.62:38797)
0926 15:04:38.174008 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:38.174023 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:38.174052 (+ 29us) server_negotiation.cc:407] Connection header received
0926 15:04:38.174906 (+ 854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:38.174916 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:38.174978 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:38.175159 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:38.176066 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:38.177412 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:38.178629 (+ 1217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:38.179154 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:38.179309 (+ 155us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:38.182233 (+ 2924us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:38.182237 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:38.182237 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:38.182250 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:38.184371 (+ 2121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:38.184933 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:38.184936 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:38.184936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:38.185069 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:38.185356 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:38.185357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:38.185357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:38.185667 (+ 310us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:38.185818 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:38.186018 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 15:04:38.186121 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":100,"threads_started":1}
I20260926 15:04:38.187342 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:04:38.195490 30241 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:38.196122 30241 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:38.197073 30225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:38.189053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44973 (local address 127.8.102.62:38797)
0926 15:04:38.189095 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:38.189097 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:38.189130 (+ 33us) server_negotiation.cc:407] Connection header received
0926 15:04:38.189202 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:38.189203 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:38.189214 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:38.189273 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:38.189900 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:38.190800 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:38.191842 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:38.192230 (+ 388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:38.192314 (+ 84us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:38.193807 (+ 1493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:38.193809 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:38.193809 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:38.193818 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:38.195642 (+ 1824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:38.196162 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:38.196164 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:38.196164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:38.196227 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:38.196475 (+ 248us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:38.196476 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:38.196476 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:38.196596 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:38.196707 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:38.196901 (+ 194us) server_negotiation.cc:299] Negotiation successful
0926 15:04:38.196979 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 15:04:38.202668 30241 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b11ef3e1a1ef4ebcbb0808942f0c6424: Generated new TSK 0
I20260926 15:04:38.202893 30241 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:38.207891 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 30153
W20260926 15:04:38.212306 30138 connection.cc:535] client connection to 127.8.102.62:38797 recv error: Network error: failed to read from TLS socket (remote: 127.8.102.62:38797): Connection reset by peer (error 104)
2026-09-26T15:04:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1375 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:38 dist-test-slave-jnhs krb5kdc[30252](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:38 dist-test-slave-jnhs krb5kdc[30252](info): set up 2 sockets
Sep 26 15:04:38 dist-test-slave-jnhs krb5kdc[30252](info): commencing operation
krb5kdc: starting...
W20260926 15:04:39.251010 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.025s user 0.004s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:39 dist-test-slave-jnhs krb5kdc[30252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:04:39.383284 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:46327
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:35685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:46327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:39.465395 30265 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:39.465687 30265 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:39.465731 30265 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:39.467053 30265 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:39.467124 30265 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:39.467151 30265 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:39.467180 30265 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:39.467201 30265 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:39.469045 30265 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:35685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:46327
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:46327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30265
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:39.469395 30265 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:39.469657 30265 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:39.474943 30273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:39.474941 30271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:39.475154 30265 server_base.cc:1034] running on GCE node
W20260926 15:04:39.475229 30270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:39.475472 30265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:39.475744 30265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:39.476949 30265 hybrid_clock.cc:648] HybridClock initialized: now 1790435079476913 us; error 49 us; skew 500 ppm
Sep 26 15:04:39 dist-test-slave-jnhs krb5kdc[30252](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435079, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:39.486806 30265 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:04:39.487350 30265 webserver.cc:460] Webserver started at http://127.8.102.62:36201/ using document root <none> and password file <none>
I20260926 15:04:39.487641 30265 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:39.487721 30265 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:39.487880 30265 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:39.489017 30265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "5a44bf572d784ba89077b55b8c39aa9e"
format_stamp: "Formatted at 2026-09-26 15:04:39 on dist-test-slave-jnhs"
server_key: "1c81d76c6537d066f169531d727d2a79"
server_key_iv: "2d23cdea4a8cb6df9a90160092dd1d60"
server_key_version: "encryptionkey@0"
I20260926 15:04:39.489401 30265 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "5a44bf572d784ba89077b55b8c39aa9e"
format_stamp: "Formatted at 2026-09-26 15:04:39 on dist-test-slave-jnhs"
server_key: "1c81d76c6537d066f169531d727d2a79"
server_key_iv: "2d23cdea4a8cb6df9a90160092dd1d60"
server_key_version: "encryptionkey@0"
I20260926 15:04:39.490942 30265 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260926 15:04:39.491954 30280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:39.492269 30265 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:39.492399 30265 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "5a44bf572d784ba89077b55b8c39aa9e"
format_stamp: "Formatted at 2026-09-26 15:04:39 on dist-test-slave-jnhs"
server_key: "1c81d76c6537d066f169531d727d2a79"
server_key_iv: "2d23cdea4a8cb6df9a90160092dd1d60"
server_key_version: "encryptionkey@0"
I20260926 15:04:39.492486 30265 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:39.514539 30265 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:39.517097 30265 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:39.517493 30265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:39.522015 30265 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:46327
I20260926 15:04:39.522058 30332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:46327 every 8 connection(s)
I20260926 15:04:39.522437 30265 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:39.523403 30333 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:39.525821 30333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e: Bootstrap starting.
I20260926 15:04:39.526539 30333 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:39.526854 30333 log.cc:824] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:39.527622 30333 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e: No bootstrap required, opened a new log
I20260926 15:04:39.528996 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 30265
I20260926 15:04:39.529125 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:39.529330 8600 external_mini_cluster.cc:1428] Setting key 36abfd464f1dfa4cdb43793758570053
I20260926 15:04:39.529732 30333 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a44bf572d784ba89077b55b8c39aa9e" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 46327 } }
I20260926 15:04:39.529944 30333 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:39.529987 30333 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5a44bf572d784ba89077b55b8c39aa9e, State: Initialized, Role: FOLLOWER
I20260926 15:04:39.530211 30333 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a44bf572d784ba89077b55b8c39aa9e" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 46327 } }
I20260926 15:04:39.530321 30333 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:39.530366 30333 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:39.530421 30333 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:39.531613 30333 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a44bf572d784ba89077b55b8c39aa9e" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 46327 } }
I20260926 15:04:39.531728 30333 leader_election.cc:302] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5a44bf572d784ba89077b55b8c39aa9e; no voters:
I20260926 15:04:39.531884 30333 leader_election.cc:288] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:39.531961 30338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:39.532114 30333 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:39.532186 30338 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e [term 1 LEADER]: Becoming Leader. State: Replica: 5a44bf572d784ba89077b55b8c39aa9e, State: Running, Role: LEADER
I20260926 15:04:39.532320 30338 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a44bf572d784ba89077b55b8c39aa9e" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 46327 } }
I20260926 15:04:39.532639 30338 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5a44bf572d784ba89077b55b8c39aa9e. Latest consensus state: current_term: 1 leader_uuid: "5a44bf572d784ba89077b55b8c39aa9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a44bf572d784ba89077b55b8c39aa9e" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 46327 } } }
I20260926 15:04:39.532665 30340 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5a44bf572d784ba89077b55b8c39aa9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5a44bf572d784ba89077b55b8c39aa9e" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 46327 } } }
I20260926 15:04:39.532779 30338 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:39.532843 30340 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e [sys.catalog]: This master's current role is: LEADER
W20260926 15:04:39.534042 30353 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:04:39.534145 30353 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:04:39.534251 30347 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:39.534466 30347 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:39.535899 30337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:39.529934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44385 (local address 127.8.102.62:46327)
0926 15:04:39.530123 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:39.530133 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:39.530170 (+ 37us) server_negotiation.cc:407] Connection header received
0926 15:04:39.530305 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:39.530311 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:39.530376 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:39.530541 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:39.530926 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:39.532361 (+ 1435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:39.533630 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:39.534086 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:39.534236 (+ 150us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:39.535087 (+ 851us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:39.535091 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:39.535091 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:39.535320 (+ 229us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:39.535358 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:39.535373 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:39.535462 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:39.535633 (+ 171us) server_negotiation.cc:299] Negotiation successful
0926 15:04:39.535665 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":103,"threads_started":1}
I20260926 15:04:39.536126 30347 catalog_manager.cc:1347] Generated new cluster ID: aebbf5d2e5f8492397e23b4254527134
I20260926 15:04:39.536176 30347 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:39.536898 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:04:39.541363 30337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:39.538251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51011 (local address 127.8.102.62:46327)
0926 15:04:39.538299 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:39.538301 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:39.538417 (+ 116us) server_negotiation.cc:407] Connection header received
0926 15:04:39.538494 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:39.538496 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:39.538507 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:39.538556 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:39.538736 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:39.539255 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:39.539964 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:39.540251 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:39.540316 (+ 65us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:39.541108 (+ 792us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:39.541110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:39.541110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:39.541115 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:39.541130 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:39.541134 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:39.541189 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:39.541251 (+ 62us) server_negotiation.cc:299] Negotiation successful
0926 15:04:39.541266 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 15:04:39.551985 30347 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:39.552575 30347 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:39.555501 30347 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5a44bf572d784ba89077b55b8c39aa9e: Generated new TSK 0
I20260926 15:04:39.555696 30347 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:39.560389 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 30265
W20260926 15:04:39.564940 30250 connection.cc:535] client connection to 127.8.102.62:46327 recv error: Network error: failed to read from TLS socket (remote: 127.8.102.62:46327): Connection reset by peer (error 104)
2026-09-26T15:04:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1352 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:39 dist-test-slave-jnhs krb5kdc[30364](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:39 dist-test-slave-jnhs krb5kdc[30364](info): set up 2 sockets
Sep 26 15:04:39 dist-test-slave-jnhs krb5kdc[30364](info): commencing operation
krb5kdc: starting...
W20260926 15:04:40.607070 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.001s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:40 dist-test-slave-jnhs krb5kdc[30364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435080, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:04:40.741767 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:44433
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:42147
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:44433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:40.824759 30377 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:40.824965 30377 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:40.824988 30377 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:40.826191 30377 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:40.826241 30377 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:40.826256 30377 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:40.826268 30377 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:40.826282 30377 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:40.827821 30377 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:42147
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:44433
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:44433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30377
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:40.828092 30377 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:40.828353 30377 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:40.834115 30385 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:40.834079 30382 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:40.834266 30377 server_base.cc:1034] running on GCE node
W20260926 15:04:40.834318 30383 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:40.834749 30377 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:40.835086 30377 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:40.836279 30377 hybrid_clock.cc:648] HybridClock initialized: now 1790435080836261 us; error 29 us; skew 500 ppm
Sep 26 15:04:40 dist-test-slave-jnhs krb5kdc[30364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435080, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:40.839120 30377 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:04:40.839627 30377 webserver.cc:460] Webserver started at http://127.8.102.62:38015/ using document root <none> and password file <none>
I20260926 15:04:40.839838 30377 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:40.839880 30377 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:40.839972 30377 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:40.841038 30377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "0c8761aa9fa7479da9951a5c5ecf43d5"
format_stamp: "Formatted at 2026-09-26 15:04:40 on dist-test-slave-jnhs"
server_key: "00482c9b755a2a2bd1fd3620e49525ff"
server_key_iv: "19c1e1c70216db46408628ecb75f0095"
server_key_version: "encryptionkey@0"
I20260926 15:04:40.841343 30377 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "0c8761aa9fa7479da9951a5c5ecf43d5"
format_stamp: "Formatted at 2026-09-26 15:04:40 on dist-test-slave-jnhs"
server_key: "00482c9b755a2a2bd1fd3620e49525ff"
server_key_iv: "19c1e1c70216db46408628ecb75f0095"
server_key_version: "encryptionkey@0"
I20260926 15:04:40.842784 30377 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 15:04:40.843667 30392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:40.844010 30377 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:40.844130 30377 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "0c8761aa9fa7479da9951a5c5ecf43d5"
format_stamp: "Formatted at 2026-09-26 15:04:40 on dist-test-slave-jnhs"
server_key: "00482c9b755a2a2bd1fd3620e49525ff"
server_key_iv: "19c1e1c70216db46408628ecb75f0095"
server_key_version: "encryptionkey@0"
I20260926 15:04:40.844244 30377 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:40.848131 30377 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:40.850474 30377 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:40.850639 30377 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:40.854722 30377 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:44433
I20260926 15:04:40.854748 30444 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:44433 every 8 connection(s)
I20260926 15:04:40.855106 30377 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:40.856267 30445 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:40.857045 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 30377
I20260926 15:04:40.857169 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:40.857373 8600 external_mini_cluster.cc:1428] Setting key 2a6206b15f700001fbd71c0acebf0fd5
I20260926 15:04:40.859606 30445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5: Bootstrap starting.
I20260926 15:04:40.860159 30448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:40.857973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36323 (local address 127.8.102.62:44433)
0926 15:04:40.858205 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:40.858223 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:40.858261 (+ 38us) server_negotiation.cc:407] Connection header received
0926 15:04:40.858757 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:40.858764 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:40.858793 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:40.858948 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:40.859707 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:40.859711 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:40.859712 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:40.859717 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:40.859765 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:40.859779 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:40.859869 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:40.859947 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 15:04:40.859992 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":80,"threads_started":1}
I20260926 15:04:40.860383 30445 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:40.860672 30445 log.cc:824] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:40.861142 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:04:40.861464 30445 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5: No bootstrap required, opened a new log
I20260926 15:04:40.863698 30445 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8761aa9fa7479da9951a5c5ecf43d5" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44433 } }
I20260926 15:04:40.863840 30448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:40.862601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51665 (local address 127.8.102.62:44433)
0926 15:04:40.862670 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:40.862672 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:40.862736 (+ 64us) server_negotiation.cc:407] Connection header received
0926 15:04:40.862820 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:40.862821 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:40.862826 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:40.862880 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:40.863626 (+ 746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:40.863628 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:40.863628 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:40.863632 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:40.863648 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:40.863653 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:40.863704 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:40.863759 (+ 55us) server_negotiation.cc:299] Negotiation successful
0926 15:04:40.863774 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 15:04:40.863847 30445 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:40.864035 30445 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0c8761aa9fa7479da9951a5c5ecf43d5, State: Initialized, Role: FOLLOWER
I20260926 15:04:40.864398 30445 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8761aa9fa7479da9951a5c5ecf43d5" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44433 } }
I20260926 15:04:40.864504 30445 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:40.864531 30445 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:40.864563 30445 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:40.865181 30445 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8761aa9fa7479da9951a5c5ecf43d5" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44433 } }
I20260926 15:04:40.865272 30445 leader_election.cc:302] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0c8761aa9fa7479da9951a5c5ecf43d5; no voters:
I20260926 15:04:40.865432 30445 leader_election.cc:288] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:40.865489 30455 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:40.865654 30455 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5 [term 1 LEADER]: Becoming Leader. State: Replica: 0c8761aa9fa7479da9951a5c5ecf43d5, State: Running, Role: LEADER
I20260926 15:04:40.865753 30445 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:40.865818 30455 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8761aa9fa7479da9951a5c5ecf43d5" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44433 } }
I20260926 15:04:40.866230 30457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0c8761aa9fa7479da9951a5c5ecf43d5. Latest consensus state: current_term: 1 leader_uuid: "0c8761aa9fa7479da9951a5c5ecf43d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8761aa9fa7479da9951a5c5ecf43d5" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44433 } } }
I20260926 15:04:40.866220 30456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0c8761aa9fa7479da9951a5c5ecf43d5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0c8761aa9fa7479da9951a5c5ecf43d5" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44433 } } }
I20260926 15:04:40.866351 30457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:40.866360 30456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:40.866771 30464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:40.867237 30464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:40.868881 30464 catalog_manager.cc:1347] Generated new cluster ID: c3f6a73c6c0d4bf49dcd7fd27ebddf34
I20260926 15:04:40.868928 30464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:40.893657 30464 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:40.893867 30464 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:40.902091 30464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0c8761aa9fa7479da9951a5c5ecf43d5: Generated new TSK 0
I20260926 15:04:40.902314 30464 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:40.910260 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 30377
2026-09-26T15:04:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:40 dist-test-slave-jnhs krb5kdc[30476](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:40 dist-test-slave-jnhs krb5kdc[30476](info): set up 2 sockets
Sep 26 15:04:40 dist-test-slave-jnhs krb5kdc[30476](info): commencing operation
krb5kdc: starting...
W20260926 15:04:41.966954 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.038s user 0.000s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:42 dist-test-slave-jnhs krb5kdc[30476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:04:42.102754 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:37123
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:44231
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:37123
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:42.186080 30489 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:42.186365 30489 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:42.186405 30489 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:42.187655 30489 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:42.187744 30489 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:42.187772 30489 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:42.187793 30489 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:42.187819 30489 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:42.189333 30489 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:44231
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:37123
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:37123
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30489
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:42.189646 30489 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:42.189883 30489 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:42.195155 30494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:42.195188 30497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:42.195194 30495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:42.195555 30489 server_base.cc:1034] running on GCE node
I20260926 15:04:42.195839 30489 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:42.196161 30489 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:42.197333 30489 hybrid_clock.cc:648] HybridClock initialized: now 1790435082197333 us; error 42 us; skew 500 ppm
Sep 26 15:04:42 dist-test-slave-jnhs krb5kdc[30476](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435082, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:42.200394 30489 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:04:42.200958 30489 webserver.cc:460] Webserver started at http://127.8.102.62:41469/ using document root <none> and password file <none>
I20260926 15:04:42.201205 30489 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:42.201292 30489 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:42.201452 30489 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:42.202575 30489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "33c87870195d413bb4442243208b564e"
format_stamp: "Formatted at 2026-09-26 15:04:42 on dist-test-slave-jnhs"
server_key: "34d45bbc55e2e29a755032c91be66e57"
server_key_iv: "a768beaf42a8f0074d4232a581ada897"
server_key_version: "encryptionkey@0"
I20260926 15:04:42.203032 30489 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "33c87870195d413bb4442243208b564e"
format_stamp: "Formatted at 2026-09-26 15:04:42 on dist-test-slave-jnhs"
server_key: "34d45bbc55e2e29a755032c91be66e57"
server_key_iv: "a768beaf42a8f0074d4232a581ada897"
server_key_version: "encryptionkey@0"
I20260926 15:04:42.204500 30489 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260926 15:04:42.205412 30504 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:42.205662 30489 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:42.205780 30489 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "33c87870195d413bb4442243208b564e"
format_stamp: "Formatted at 2026-09-26 15:04:42 on dist-test-slave-jnhs"
server_key: "34d45bbc55e2e29a755032c91be66e57"
server_key_iv: "a768beaf42a8f0074d4232a581ada897"
server_key_version: "encryptionkey@0"
I20260926 15:04:42.205871 30489 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:42.219156 30489 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:42.221465 30489 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:42.221671 30489 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:42.226239 30489 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:37123
I20260926 15:04:42.226377 30556 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:37123 every 8 connection(s)
I20260926 15:04:42.226837 30489 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:42.227916 30557 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:42.228384 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 30489
I20260926 15:04:42.228545 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:42.228811 8600 external_mini_cluster.cc:1428] Setting key 1efe71967fc8c8b05f7a18e331cc447d
I20260926 15:04:42.231245 30557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e: Bootstrap starting.
I20260926 15:04:42.231968 30557 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:42.232301 30557 log.cc:824] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:42.233275 30557 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e: No bootstrap required, opened a new log
I20260926 15:04:42.235646 30557 raft_consensus.cc:348] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c87870195d413bb4442243208b564e" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37123 } }
I20260926 15:04:42.235808 30557 raft_consensus.cc:374] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:42.235832 30557 raft_consensus.cc:729] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c87870195d413bb4442243208b564e, State: Initialized, Role: FOLLOWER
I20260926 15:04:42.236022 30557 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c87870195d413bb4442243208b564e" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37123 } }
I20260926 15:04:42.236129 30557 raft_consensus.cc:388] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:42.236155 30557 raft_consensus.cc:482] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:42.236215 30557 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e [term 0 FOLLOWER]: Advancing to term 1
Sep 26 15:04:42 dist-test-slave-jnhs krb5kdc[30476](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:04:42.236925 30557 raft_consensus.cc:504] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c87870195d413bb4442243208b564e" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37123 } }
I20260926 15:04:42.237073 30557 leader_election.cc:302] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33c87870195d413bb4442243208b564e; no voters:
I20260926 15:04:42.237272 30557 leader_election.cc:288] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:42.237365 30562 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:42.237555 30562 raft_consensus.cc:686] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e [term 1 LEADER]: Becoming Leader. State: Replica: 33c87870195d413bb4442243208b564e, State: Running, Role: LEADER
I20260926 15:04:42.237574 30557 sys_catalog.cc:564] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:42.237740 30562 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c87870195d413bb4442243208b564e" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37123 } }
I20260926 15:04:42.238143 30564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33c87870195d413bb4442243208b564e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c87870195d413bb4442243208b564e" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37123 } } }
I20260926 15:04:42.238233 30564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:42.238503 30570 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:42.239078 30570 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:42.239219 30563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33c87870195d413bb4442243208b564e. Latest consensus state: current_term: 1 leader_uuid: "33c87870195d413bb4442243208b564e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33c87870195d413bb4442243208b564e" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 37123 } } }
I20260926 15:04:42.239563 30563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:42.240990 30570 catalog_manager.cc:1347] Generated new cluster ID: 90f9a6a36d364835b012e44b29223eb2
I20260926 15:04:42.241039 30570 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:42.241308 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:42.229450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46327 (local address 127.8.102.62:37123)
0926 15:04:42.229653 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:42.229664 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:42.229702 (+ 38us) server_negotiation.cc:407] Connection header received
0926 15:04:42.230257 (+ 555us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:42.230267 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:42.230340 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:42.230510 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:42.231174 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:42.232584 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:42.233546 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:42.234161 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:42.234290 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:42.237098 (+ 2808us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:42.237102 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:42.237102 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:42.237117 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:42.239405 (+ 2288us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:42.239846 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:42.239850 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:42.239851 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:42.239961 (+ 110us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:42.240174 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:42.240175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:42.240175 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:42.240513 (+ 338us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:42.240669 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:42.240907 (+ 238us) server_negotiation.cc:299] Negotiation successful
0926 15:04:42.241075 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":71,"threads_started":1}
I20260926 15:04:42.242272 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:04:42.248863 30570 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:42.249565 30570 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:42.250428 30560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:42.243838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48041 (local address 127.8.102.62:37123)
0926 15:04:42.243876 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:42.243878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:42.243923 (+ 45us) server_negotiation.cc:407] Connection header received
0926 15:04:42.243999 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:42.243999 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:42.244010 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:42.244057 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:42.244655 (+ 598us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:42.245148 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:42.245881 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:42.246116 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:42.246203 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:42.247608 (+ 1405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:42.247610 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:42.247610 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:42.247619 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:42.249183 (+ 1564us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:42.249562 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:42.249563 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:42.249564 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:42.249655 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:42.249870 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:42.249871 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:42.249871 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:42.249988 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:42.250111 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:42.250216 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 15:04:42.250310 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260926 15:04:42.256805 30570 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 33c87870195d413bb4442243208b564e: Generated new TSK 0
I20260926 15:04:42.257009 30570 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:42.260955 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 30489
2026-09-26T15:04:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1349 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:42 dist-test-slave-jnhs krb5kdc[30588](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:42 dist-test-slave-jnhs krb5kdc[30588](info): set up 2 sockets
Sep 26 15:04:42 dist-test-slave-jnhs krb5kdc[30588](info): commencing operation
krb5kdc: starting...
W20260926 15:04:43.306964 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.028s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:43 dist-test-slave-jnhs krb5kdc[30588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435083, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:04:43.442435 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:38097
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:39171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:38097
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:43.525591 30601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:43.525884 30601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:43.525926 30601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:43.527223 30601 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:43.527314 30601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:43.527339 30601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:43.527362 30601 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:43.527391 30601 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:43.528936 30601 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:39171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:38097
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:38097
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:43.529251 30601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:43.529500 30601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:43.534904 30609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:43.534898 30601 server_base.cc:1034] running on GCE node
W20260926 15:04:43.534898 30607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:43.534895 30606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:43.535516 30601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:43.535869 30601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:43.537020 30601 hybrid_clock.cc:648] HybridClock initialized: now 1790435083537008 us; error 46 us; skew 500 ppm
Sep 26 15:04:43 dist-test-slave-jnhs krb5kdc[30588](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435083, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:43.540043 30601 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:04:43.540616 30601 webserver.cc:460] Webserver started at http://127.8.102.62:38125/ using document root <none> and password file <none>
I20260926 15:04:43.540879 30601 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:43.540956 30601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:43.541115 30601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:43.542363 30601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "f9c720f53794426fa0727d8c73c3f1ca"
format_stamp: "Formatted at 2026-09-26 15:04:43 on dist-test-slave-jnhs"
server_key: "2d931aa0eaf2c7e1865c112a4a0395d4"
server_key_iv: "0c3b8a5d8cd3a8639c08e09fbef2f1ad"
server_key_version: "encryptionkey@0"
I20260926 15:04:43.542824 30601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "f9c720f53794426fa0727d8c73c3f1ca"
format_stamp: "Formatted at 2026-09-26 15:04:43 on dist-test-slave-jnhs"
server_key: "2d931aa0eaf2c7e1865c112a4a0395d4"
server_key_iv: "0c3b8a5d8cd3a8639c08e09fbef2f1ad"
server_key_version: "encryptionkey@0"
I20260926 15:04:43.544551 30601 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.000s sys 0.002s
I20260926 15:04:43.545395 30616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:43.545609 30601 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260926 15:04:43.545722 30601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "f9c720f53794426fa0727d8c73c3f1ca"
format_stamp: "Formatted at 2026-09-26 15:04:43 on dist-test-slave-jnhs"
server_key: "2d931aa0eaf2c7e1865c112a4a0395d4"
server_key_iv: "0c3b8a5d8cd3a8639c08e09fbef2f1ad"
server_key_version: "encryptionkey@0"
I20260926 15:04:43.545815 30601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:43.555401 30601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:43.557855 30601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:43.558053 30601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:43.562371 30601 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:38097
I20260926 15:04:43.562440 30668 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:38097 every 8 connection(s)
I20260926 15:04:43.562865 30601 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:43.563848 30669 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:43.566197 30669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca: Bootstrap starting.
I20260926 15:04:43.566870 30669 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:43.567111 30669 log.cc:824] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:43.567688 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 30601
I20260926 15:04:43.567818 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:43.567938 30669 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca: No bootstrap required, opened a new log
I20260926 15:04:43.568051 8600 external_mini_cluster.cc:1428] Setting key 07b9308ac0d8edcbac763b006029bffe
I20260926 15:04:43.570770 30669 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c720f53794426fa0727d8c73c3f1ca" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38097 } }
I20260926 15:04:43.570995 30669 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:43.571034 30669 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9c720f53794426fa0727d8c73c3f1ca, State: Initialized, Role: FOLLOWER
I20260926 15:04:43.571163 30669 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c720f53794426fa0727d8c73c3f1ca" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38097 } }
I20260926 15:04:43.571271 30669 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:43.571305 30669 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:43.571352 30669 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:43.572072 30669 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c720f53794426fa0727d8c73c3f1ca" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38097 } }
I20260926 15:04:43.572211 30669 leader_election.cc:302] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9c720f53794426fa0727d8c73c3f1ca; no voters:
I20260926 15:04:43.572450 30669 leader_election.cc:288] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:43.572476 30674 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:43.572690 30674 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca [term 1 LEADER]: Becoming Leader. State: Replica: f9c720f53794426fa0727d8c73c3f1ca, State: Running, Role: LEADER
I20260926 15:04:43.572836 30669 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:43.573199 30674 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c720f53794426fa0727d8c73c3f1ca" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38097 } }
I20260926 15:04:43.573585 30675 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9c720f53794426fa0727d8c73c3f1ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c720f53794426fa0727d8c73c3f1ca" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38097 } } }
I20260926 15:04:43.573700 30675 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:43.573601 30676 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9c720f53794426fa0727d8c73c3f1ca. Latest consensus state: current_term: 1 leader_uuid: "f9c720f53794426fa0727d8c73c3f1ca" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9c720f53794426fa0727d8c73c3f1ca" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 38097 } } }
I20260926 15:04:43.573954 30676 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:43.574251 30673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:43.568663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55833 (local address 127.8.102.62:38097)
0926 15:04:43.568833 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:43.568841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:43.568867 (+ 26us) server_negotiation.cc:407] Connection header received
0926 15:04:43.569351 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:43.569358 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:43.569406 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:43.569524 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:43.569787 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:43.570738 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:43.571832 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:43.572398 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:43.572520 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:43.573297 (+ 777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:43.573301 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:43.573301 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:43.573517 (+ 216us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:43.573559 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:43.573576 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:43.573676 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:43.573973 (+ 297us) server_negotiation.cc:299] Negotiation successful
0926 15:04:43.574053 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":86,"threads_started":1}
W20260926 15:04:43.575066 30689 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:04:43.575124 30689 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:04:43.575172 30682 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:43.575251 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:04:43.575352 30682 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:43.577077 30682 catalog_manager.cc:1347] Generated new cluster ID: c18769995f28497f95de90775824cd34
I20260926 15:04:43.577134 30682 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:43.580765 30673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:43.576882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48199 (local address 127.8.102.62:38097)
0926 15:04:43.576918 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:43.576920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:43.576982 (+ 62us) server_negotiation.cc:407] Connection header received
0926 15:04:43.577078 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:43.577079 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:43.577089 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:43.577122 (+ 33us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:43.577367 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:43.578197 (+ 830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:43.579290 (+ 1093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:43.579624 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:43.579703 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:43.580466 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:43.580467 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:43.580467 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:43.580472 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:43.580487 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:43.580492 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:43.580548 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:43.580650 (+ 102us) server_negotiation.cc:299] Negotiation successful
0926 15:04:43.580673 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":18}
I20260926 15:04:43.603562 30682 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:43.604357 30682 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:43.610001 30682 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9c720f53794426fa0727d8c73c3f1ca: Generated new TSK 0
I20260926 15:04:43.610220 30682 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:43.614670 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 30601
2026-09-26T15:04:43Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1354 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:43 dist-test-slave-jnhs krb5kdc[30700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:43 dist-test-slave-jnhs krb5kdc[30700](info): set up 2 sockets
Sep 26 15:04:43 dist-test-slave-jnhs krb5kdc[30700](info): commencing operation
krb5kdc: starting...
W20260926 15:04:44.711023 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.078s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:44 dist-test-slave-jnhs krb5kdc[30700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435084, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:04:44.845077 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:42997
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:37841
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:42997
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:44.927997 30713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:44.928246 30713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:44.928272 30713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:44.929530 30713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:44.929584 30713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:44.929598 30713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:44.929611 30713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:44.929623 30713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:44.931236 30713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:37841
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:42997
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:42997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30713
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:44.931532 30713 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:44.931785 30713 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:44.937757 30718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:44.938239 30719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:44.938202 30721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:44.938200 30713 server_base.cc:1034] running on GCE node
I20260926 15:04:44.938915 30713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:44.939383 30713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:44.940593 30713 hybrid_clock.cc:648] HybridClock initialized: now 1790435084940545 us; error 46 us; skew 500 ppm
Sep 26 15:04:44 dist-test-slave-jnhs krb5kdc[30700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435084, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:44.944451 30713 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:04:44.945144 30713 webserver.cc:460] Webserver started at http://127.8.102.62:36053/ using document root <none> and password file <none>
I20260926 15:04:44.945489 30713 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:44.945557 30713 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:44.945711 30713 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:44.947331 30713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "b404f9d8f4594d87bd6ac14093053508"
format_stamp: "Formatted at 2026-09-26 15:04:44 on dist-test-slave-jnhs"
server_key: "eae91cd8dd6103679a0817fa72e555ef"
server_key_iv: "81ce365504540d04c3292d1e6267420f"
server_key_version: "encryptionkey@0"
I20260926 15:04:44.947819 30713 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "b404f9d8f4594d87bd6ac14093053508"
format_stamp: "Formatted at 2026-09-26 15:04:44 on dist-test-slave-jnhs"
server_key: "eae91cd8dd6103679a0817fa72e555ef"
server_key_iv: "81ce365504540d04c3292d1e6267420f"
server_key_version: "encryptionkey@0"
I20260926 15:04:44.949579 30713 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.001s
I20260926 15:04:44.950718 30728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:44.950932 30713 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:44.951073 30713 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "b404f9d8f4594d87bd6ac14093053508"
format_stamp: "Formatted at 2026-09-26 15:04:44 on dist-test-slave-jnhs"
server_key: "eae91cd8dd6103679a0817fa72e555ef"
server_key_iv: "81ce365504540d04c3292d1e6267420f"
server_key_version: "encryptionkey@0"
I20260926 15:04:44.951206 30713 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:44.956951 30713 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:44.960044 30713 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:44.960297 30713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:44.965850 30713 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:42997
I20260926 15:04:44.965914 30780 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:42997 every 8 connection(s)
I20260926 15:04:44.966331 30713 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:44.967554 30781 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:44.970531 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 30713
I20260926 15:04:44.970680 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:44.970854 30781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508: Bootstrap starting.
I20260926 15:04:44.970880 8600 external_mini_cluster.cc:1428] Setting key c0c336f2f74b294db0223dd058cf7fc5
I20260926 15:04:44.971599 30781 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:44.971935 30781 log.cc:824] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:44.972779 30781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508: No bootstrap required, opened a new log
I20260926 15:04:44.975582 30781 raft_consensus.cc:348] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b404f9d8f4594d87bd6ac14093053508" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 42997 } }
I20260926 15:04:44.975778 30781 raft_consensus.cc:374] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260926 15:04:44.975826 30781 raft_consensus.cc:729] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b404f9d8f4594d87bd6ac14093053508, State: Initialized, Role: FOLLOWER
I20260926 15:04:44.975963 30781 consensus_queue.cc:260] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b404f9d8f4594d87bd6ac14093053508" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 42997 } }
I20260926 15:04:44.976018 30784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:44.971620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35685 (local address 127.8.102.62:42997)
0926 15:04:44.971813 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:44.971822 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:44.971872 (+ 50us) server_negotiation.cc:407] Connection header received
0926 15:04:44.972593 (+ 721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:44.972604 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:44.972646 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:44.972870 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:44.975300 (+ 2430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:44.975305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:44.975305 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:44.975311 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:44.975368 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:44.975401 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:44.975500 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:44.975672 (+ 172us) server_negotiation.cc:299] Negotiation successful
0926 15:04:44.975719 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260926 15:04:44.976096 30781 raft_consensus.cc:388] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260926 15:04:44.976212 30781 raft_consensus.cc:482] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260926 15:04:44.976281 30781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:44.977099 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:04:44.977114 30781 raft_consensus.cc:504] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b404f9d8f4594d87bd6ac14093053508" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 42997 } }
I20260926 15:04:44.977221 30781 leader_election.cc:302] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b404f9d8f4594d87bd6ac14093053508; no voters:
I20260926 15:04:44.977392 30781 leader_election.cc:288] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260926 15:04:44.977435 30786 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:44.977598 30786 raft_consensus.cc:686] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508 [term 1 LEADER]: Becoming Leader. State: Replica: b404f9d8f4594d87bd6ac14093053508, State: Running, Role: LEADER
I20260926 15:04:44.977674 30781 sys_catalog.cc:564] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:44.977702 30786 consensus_queue.cc:237] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b404f9d8f4594d87bd6ac14093053508" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 42997 } }
I20260926 15:04:44.978075 30786 sys_catalog.cc:455] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b404f9d8f4594d87bd6ac14093053508. Latest consensus state: current_term: 1 leader_uuid: "b404f9d8f4594d87bd6ac14093053508" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b404f9d8f4594d87bd6ac14093053508" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 42997 } } }
I20260926 15:04:44.978159 30786 sys_catalog.cc:458] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:44.978087 30788 sys_catalog.cc:455] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b404f9d8f4594d87bd6ac14093053508" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b404f9d8f4594d87bd6ac14093053508" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 42997 } } }
I20260926 15:04:44.978436 30788 sys_catalog.cc:458] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508 [sys.catalog]: This master's current role is: LEADER
W20260926 15:04:44.979781 30806 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:04:44.979843 30806 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:04:44.979908 30795 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:44.980096 30795 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:44.981113 30784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:44.978712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45729 (local address 127.8.102.62:42997)
0926 15:04:44.978752 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:44.978754 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:44.979548 (+ 794us) server_negotiation.cc:407] Connection header received
0926 15:04:44.979613 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:44.979614 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:44.979619 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:44.979684 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:44.980884 (+ 1200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:44.980885 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:44.980886 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0926 15:04:44.980890 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:44.980904 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0926 15:04:44.980909 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:44.980937 (+ 28us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:44.981042 (+ 105us) server_negotiation.cc:299] Negotiation successful
0926 15:04:44.981056 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 15:04:44.981799 30795 catalog_manager.cc:1347] Generated new cluster ID: cd8f966cbc9c4233a8e1a040187da1a6
I20260926 15:04:44.981899 30795 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:45.006711 30795 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:45.006973 30795 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:45.024724 30795 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b404f9d8f4594d87bd6ac14093053508: Generated new TSK 0
I20260926 15:04:45.024981 30795 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260926 15:04:45.028632 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 30713
2026-09-26T15:04:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1414 ms)
[----------] 18 tests from AuthTokenIssuingTest (24453 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:45 dist-test-slave-jnhs krb5kdc[30812](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:45 dist-test-slave-jnhs krb5kdc[30812](info): set up 2 sockets
Sep 26 15:04:45 dist-test-slave-jnhs krb5kdc[30812](info): commencing operation
krb5kdc: starting...
W20260926 15:04:46.078985 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.032s user 0.005s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:46 dist-test-slave-jnhs krb5kdc[30812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:04:46.211117 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:36159
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:36159,127.8.102.61:34855,127.8.102.60:42155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:46.299137 30825 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 15:04:46.299424 30825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:46.299456 30825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:46.299499 30825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:46.300701 30825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:46.300792 30825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:46.300818 30825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:46.300848 30825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:46.300879 30825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:46.302469 30825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:36159,127.8.102.61:34855,127.8.102.60:42155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:36159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30825
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:46.302819 30825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:46.303081 30825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:46.308269 30831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:46.308274 30830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:46.308269 30833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:46.308501 30825 server_base.cc:1034] running on GCE node
I20260926 15:04:46.308792 30825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:46.309154 30825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:46.310319 30825 hybrid_clock.cc:648] HybridClock initialized: now 1790435086310293 us; error 39 us; skew 500 ppm
Sep 26 15:04:46 dist-test-slave-jnhs krb5kdc[30812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435086, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:46.313099 30825 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:04:46.313616 30825 webserver.cc:460] Webserver started at http://127.8.102.62:34649/ using document root <none> and password file <none>
I20260926 15:04:46.313881 30825 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:46.313956 30825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:46.314108 30825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:46.315336 30825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "c94f523df154457cb556cd8703eadbc0"
format_stamp: "Formatted at 2026-09-26 15:04:46 on dist-test-slave-jnhs"
server_key: "36e6db98bf8c2af603ad57d6030d3f83"
server_key_iv: "f63d7cb3150ca77526737072280f9481"
server_key_version: "encryptionkey@0"
I20260926 15:04:46.315722 30825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "c94f523df154457cb556cd8703eadbc0"
format_stamp: "Formatted at 2026-09-26 15:04:46 on dist-test-slave-jnhs"
server_key: "36e6db98bf8c2af603ad57d6030d3f83"
server_key_iv: "f63d7cb3150ca77526737072280f9481"
server_key_version: "encryptionkey@0"
I20260926 15:04:46.317167 30825 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260926 15:04:46.318010 30840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:46.318305 30825 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:46.318435 30825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "c94f523df154457cb556cd8703eadbc0"
format_stamp: "Formatted at 2026-09-26 15:04:46 on dist-test-slave-jnhs"
server_key: "36e6db98bf8c2af603ad57d6030d3f83"
server_key_iv: "f63d7cb3150ca77526737072280f9481"
server_key_version: "encryptionkey@0"
I20260926 15:04:46.318532 30825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:46.327649 30825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:46.329854 30825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:46.330035 30825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:46.334527 30825 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:36159
I20260926 15:04:46.334712 30892 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:36159 every 8 connection(s)
I20260926 15:04:46.335042 30825 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:46.336115 30893 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:46.336517 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 30825
I20260926 15:04:46.336611 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:46.336838 8600 external_mini_cluster.cc:1428] Setting key 1cccf1b295a600dc29877dfc292715a9
I20260926 15:04:46.337944 30893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.8.102.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.61@KRBTEST.COM" created.
Sep 26 15:04:46 dist-test-slave-jnhs krb5kdc[30812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435086, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:04:46.353081 30896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.338745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47008 (local address 127.8.102.62:36159)
0926 15:04:46.338966 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.338978 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.339205 (+ 227us) server_negotiation.cc:407] Connection header received
0926 15:04:46.340040 (+ 835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.340042 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.340068 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.340211 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.341101 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.341989 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.343492 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.343854 (+ 362us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.343948 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.349226 (+ 5278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:46.349236 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:46.349237 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:46.349247 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:46.351116 (+ 1869us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.351654 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.351657 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.351657 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.351740 (+ 83us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.352039 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.352040 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.352040 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.352311 (+ 271us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:46.352376 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:46.352693 (+ 317us) server_negotiation.cc:299] Negotiation successful
0926 15:04:46.352898 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":154,"thread_start_us":87,"threads_started":1}
I20260926 15:04:46.353091 30897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.338745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36159 (local address 127.0.0.1:47008)
0926 15:04:46.339050 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.339094 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:46.339852 (+ 758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:46.340257 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:46.340260 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:46.340674 (+ 414us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:46.340962 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:46.340969 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.342110 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.342111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:46.343367 (+ 1256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:46.343371 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.343981 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.343982 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:46.344064 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.344702 (+ 638us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:46.344713 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:46.349112 (+ 4399us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:46.351226 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:46.351234 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:46.351237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:46.351534 (+ 297us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:46.351817 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:46.351818 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:46.351819 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:46.351948 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:46.352396 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:46.352404 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:46.352492 (+ 88us) client_negotiation.cc:769] Sending connection context
0926 15:04:46.352600 (+ 108us) client_negotiation.cc:240] Negotiation successful
0926 15:04:46.352811 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":272,"thread_start_us":91,"threads_started":1}
I20260926 15:04:46.354019 30893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:04:46.354444 30897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.354296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:34855 (local address 127.0.0.1:32922)
0926 15:04:46.354356 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.354412 (+ 56us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.61:34855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 15:04:46.354612 30843 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.102.61:34855: connect: Connection refused (error 111)
I20260926 15:04:46.355186 30897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.355059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:34855 (local address 127.0.0.1:32934)
0926 15:04:46.355117 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.355156 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.61:34855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 15:04:46.355382 30893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.61:34855: Network error: Client connection negotiation failed: client connection to 127.8.102.61:34855: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.61.keytab.
Entry for principal kudu/127.8.102.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.61.keytab.
I20260926 15:04:46.367266 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.61:34855
--webserver_interface=127.8.102.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.61.keytab
--principal=kudu/127.8.102.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:36159,127.8.102.61:34855,127.8.102.60:42155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:04:46.404666 30893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } attempt: 1
I20260926 15:04:46.405241 30897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.405087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:34855 (local address 127.0.0.1:32942)
0926 15:04:46.405165 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.405207 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.61:34855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20260926 15:04:46.405665 30897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.405533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:34855 (local address 127.0.0.1:32954)
0926 15:04:46.405593 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.405636 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.61:34855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 15:04:46.405804 30893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.61:34855: Network error: Client connection negotiation failed: client connection to 127.8.102.61:34855: connect: Connection refused (error 111)
W20260926 15:04:46.449908 30900 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 15:04:46.450125 30900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:46.450142 30900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:46.450163 30900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:46.451370 30900 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:46.451414 30900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:46.451429 30900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:46.451442 30900 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:46.451454 30900 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:46.453080 30900 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:36159,127.8.102.61:34855,127.8.102.60:42155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.61.keytab
--principal=kudu/127.8.102.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.61:34855
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.102.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30900
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:46.453364 30900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:46.453605 30900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:46.458963 30908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:46.458966 30905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:46.458997 30900 server_base.cc:1034] running on GCE node
W20260926 15:04:46.458963 30906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:46.459604 30900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:46.459930 30900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:46.461092 30900 hybrid_clock.cc:648] HybridClock initialized: now 1790435086461048 us; error 60 us; skew 500 ppm
Sep 26 15:04:46 dist-test-slave-jnhs krb5kdc[30812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435086, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:46.464010 30900 init.cc:377] Logged in from keytab as kudu/127.8.102.61@KRBTEST.COM (short username kudu)
I20260926 15:04:46.464597 30900 webserver.cc:460] Webserver started at http://127.8.102.61:39745/ using document root <none> and password file <none>
I20260926 15:04:46.464897 30900 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:46.464982 30900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:46.465216 30900 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:46.466444 30900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-1/data/instance:
uuid: "9f1e411dd090483aae8dd1d6a2570997"
format_stamp: "Formatted at 2026-09-26 15:04:46 on dist-test-slave-jnhs"
server_key: "c6d8a6e2b306a1247c9c98fccae7bd89"
server_key_iv: "d3ad5cd319778e6899636cd257210e09"
server_key_version: "encryptionkey@0"
I20260926 15:04:46.466902 30900 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-1/wal/instance:
uuid: "9f1e411dd090483aae8dd1d6a2570997"
format_stamp: "Formatted at 2026-09-26 15:04:46 on dist-test-slave-jnhs"
server_key: "c6d8a6e2b306a1247c9c98fccae7bd89"
server_key_iv: "d3ad5cd319778e6899636cd257210e09"
server_key_version: "encryptionkey@0"
I20260926 15:04:46.468487 30900 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:46.469363 30915 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:46.469642 30900 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:46.469790 30900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-1/wal
uuid: "9f1e411dd090483aae8dd1d6a2570997"
format_stamp: "Formatted at 2026-09-26 15:04:46 on dist-test-slave-jnhs"
server_key: "c6d8a6e2b306a1247c9c98fccae7bd89"
server_key_iv: "d3ad5cd319778e6899636cd257210e09"
server_key_version: "encryptionkey@0"
I20260926 15:04:46.469880 30900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:46.473970 30893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } attempt: 2
I20260926 15:04:46.474429 30897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.474280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:34855 (local address 127.0.0.1:32956)
0926 15:04:46.474345 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.474390 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.61:34855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 15:04:46.474826 30897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.474732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:34855 (local address 127.0.0.1:32966)
0926 15:04:46.474777 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.474805 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.61:34855: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260926 15:04:46.474946 30893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.61:34855: Network error: Client connection negotiation failed: client connection to 127.8.102.61:34855: connect: Connection refused (error 111)
I20260926 15:04:46.496099 30900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:46.498436 30900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:46.498621 30900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:46.503254 30900 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.61:34855
I20260926 15:04:46.503296 30967 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.61:34855 every 8 connection(s)
I20260926 15:04:46.503688 30900 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-1/data/info.pb
I20260926 15:04:46.504695 30968 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:46.506299 30968 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:04:46 dist-test-slave-jnhs krb5kdc[30812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435086, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.61@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:04:46.513232 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 30900
I20260926 15:04:46.513351 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-1/wal/instance
I20260926 15:04:46.513536 8600 external_mini_cluster.cc:1428] Setting key ecf28cc8992c8b0e56b6b2d6e0cd97a3
I20260926 15:04:46.515522 30896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.507046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47018 (local address 127.8.102.62:36159)
0926 15:04:46.507150 (+ 104us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.507152 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.507313 (+ 161us) server_negotiation.cc:407] Connection header received
0926 15:04:46.507772 (+ 459us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.507773 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.507786 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.507852 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.508484 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.508981 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.509973 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.510228 (+ 255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.510303 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.512536 (+ 2233us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:46.512537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:46.512538 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:46.512544 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:46.513975 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.514468 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.514469 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.514469 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.514519 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.514853 (+ 334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.514854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.514854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.514963 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:46.515050 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:46.515341 (+ 291us) server_negotiation.cc:299] Negotiation successful
0926 15:04:46.515429 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260926 15:04:46.515753 30970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.506997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36159 (local address 127.0.0.1:47018)
0926 15:04:46.507178 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.507224 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:46.507670 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:46.507945 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:46.507948 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:46.508194 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:46.508395 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:46.508399 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.509067 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.509067 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:46.509869 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:46.509871 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.510339 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.510340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:46.510442 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.511015 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:46.511023 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:46.512446 (+ 1423us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:46.514055 (+ 1609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:46.514063 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:46.514066 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:46.514392 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:46.514605 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:46.514606 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:46.514607 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:46.514786 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:46.515043 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:46.515051 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:46.515194 (+ 143us) client_negotiation.cc:769] Sending connection context
0926 15:04:46.515321 (+ 127us) client_negotiation.cc:240] Negotiation successful
0926 15:04:46.515480 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":99,"threads_started":1}
I20260926 15:04:46.516368 30968 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:04:46 dist-test-slave-jnhs krb5kdc[30812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435086, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.61@KRBTEST.COM for kudu/127.8.102.61@KRBTEST.COM
WARNING: no policy specified for kudu/127.8.102.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.60@KRBTEST.COM" created.
I20260926 15:04:46.524821 30970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.516669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:34855 (local address 127.0.0.1:32968)
0926 15:04:46.516729 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.516742 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:46.516787 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:46.517015 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:46.517016 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:46.517192 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:46.517257 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:46.517259 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.518222 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.518223 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:46.519186 (+ 963us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:46.519189 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.519700 (+ 511us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.519701 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:46.519785 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.520202 (+ 417us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:46.520207 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:46.521574 (+ 1367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:46.523778 (+ 2204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:46.523778 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:46.523779 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:46.523946 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:46.524177 (+ 231us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:46.524178 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:46.524178 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:46.524232 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:46.524546 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:46.524546 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:46.524586 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 15:04:46.524650 (+ 64us) client_negotiation.cc:240] Negotiation successful
0926 15:04:46.524752 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20260926 15:04:46.525205 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.516707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32968 (local address 127.8.102.61:34855)
0926 15:04:46.516866 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.516871 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.516889 (+ 18us) server_negotiation.cc:407] Connection header received
0926 15:04:46.516918 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.516919 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.516930 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.516990 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.517355 (+ 365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.518120 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.519293 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.519605 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.519673 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.521675 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:46.521684 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:46.521684 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:46.521698 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:46.523673 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.524042 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.524045 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.524045 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.524107 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.524305 (+ 198us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.524306 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.524306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.524469 (+ 163us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:46.524571 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:46.524998 (+ 427us) server_negotiation.cc:299] Negotiation successful
0926 15:04:46.525084 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":104,"threads_started":1}
I20260926 15:04:46.525943 30968 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:04:46.526371 30970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.526231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:42155 (local address 127.0.0.1:44230)
0926 15:04:46.526292 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.526342 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260926 15:04:46.526511 30917 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.102.60:42155: connect: Connection refused (error 111)
I20260926 15:04:46.526962 30970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.526867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:42155 (local address 127.0.0.1:44244)
0926 15:04:46.526917 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.526941 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 15:04:46.527071 30968 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.60:42155: Network error: Client connection negotiation failed: client connection to 127.8.102.60:42155: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.60.keytab.
Entry for principal kudu/127.8.102.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.60.keytab.
I20260926 15:04:46.542757 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.60:42155
--webserver_interface=127.8.102.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.60.keytab
--principal=kudu/127.8.102.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:38875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:36159,127.8.102.61:34855,127.8.102.60:42155
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:04:46.566111 30893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } attempt: 3
Sep 26 15:04:46 dist-test-slave-jnhs krb5kdc[30812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435086, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for kudu/127.8.102.61@KRBTEST.COM
I20260926 15:04:46.574043 30897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.566587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:34855 (local address 127.0.0.1:32974)
0926 15:04:46.566711 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.566724 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:46.566823 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:46.566989 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:46.566990 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:46.567173 (+ 183us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:46.567242 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:46.567244 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.567940 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.567941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:46.568486 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:46.568487 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.568914 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.568915 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:46.568999 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.569502 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:46.569513 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:46.570809 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:46.572736 (+ 1927us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:46.572737 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:46.572737 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:46.572947 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:46.573207 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:46.573208 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:46.573209 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:46.573279 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:46.573676 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:46.573676 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:46.573722 (+ 46us) client_negotiation.cc:769] Sending connection context
0926 15:04:46.573816 (+ 94us) client_negotiation.cc:240] Negotiation successful
0926 15:04:46.573932 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93}
I20260926 15:04:46.574052 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.566740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32974 (local address 127.8.102.61:34855)
0926 15:04:46.566814 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.566816 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.566828 (+ 12us) server_negotiation.cc:407] Connection header received
0926 15:04:46.566889 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.566890 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.566903 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.566984 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.567338 (+ 354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.567844 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.568584 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.568816 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.568914 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.570906 (+ 1992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:46.570908 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:46.570908 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:46.570917 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:46.572613 (+ 1696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.573059 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.573060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.573060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.573127 (+ 67us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.573357 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.573358 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.573358 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.573495 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:46.573587 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:46.573843 (+ 256us) server_negotiation.cc:299] Negotiation successful
0926 15:04:46.573932 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:04:46.574812 30893 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:04:46.575220 30897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.575093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:42155 (local address 127.0.0.1:44248)
0926 15:04:46.575145 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.575183 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260926 15:04:46.575562 30897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.575465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:42155 (local address 127.0.0.1:44252)
0926 15:04:46.575516 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.575540 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20260926 15:04:46.575660 30893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.60:42155: Network error: Client connection negotiation failed: client connection to 127.8.102.60:42155: connect: Connection refused (error 111)
I20260926 15:04:46.576273 30968 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } attempt: 1
I20260926 15:04:46.576643 30970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.576508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:42155 (local address 127.0.0.1:44266)
0926 15:04:46.576567 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.576612 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 15:04:46.576972 30970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.576873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:42155 (local address 127.0.0.1:44274)
0926 15:04:46.576924 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.576952 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260926 15:04:46.577075 30968 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.60:42155: Network error: Client connection negotiation failed: client connection to 127.8.102.60:42155: connect: Connection refused (error 111)
I20260926 15:04:46.606806 30893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } attempt: 1
I20260926 15:04:46.607342 30897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.607191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:42155 (local address 127.0.0.1:44284)
0926 15:04:46.607269 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.607312 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20260926 15:04:46.607800 30897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.607614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:42155 (local address 127.0.0.1:44286)
0926 15:04:46.607681 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.607753 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20260926 15:04:46.607985 30893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.60:42155: Network error: Client connection negotiation failed: client connection to 127.8.102.60:42155: connect: Connection refused (error 111)
W20260926 15:04:46.629154 30975 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 15:04:46.629375 30975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:46.629393 30975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:46.629415 30975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:46.630606 30975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:46.630733 30975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:46.630751 30975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:46.630764 30975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:46.630777 30975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:46.632344 30975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:38875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:36159,127.8.102.61:34855,127.8.102.60:42155
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.60.keytab
--principal=kudu/127.8.102.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.60:42155
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.102.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:46.632628 30975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:46.632905 30975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:46.638319 30983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:46.638331 30981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:46.638329 30980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:46.638463 30975 server_base.cc:1034] running on GCE node
I20260926 15:04:46.638854 30975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:46.639129 30975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:46.640303 30975 hybrid_clock.cc:648] HybridClock initialized: now 1790435086640260 us; error 57 us; skew 500 ppm
Sep 26 15:04:46 dist-test-slave-jnhs krb5kdc[30812](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435086, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:46.642980 30975 init.cc:377] Logged in from keytab as kudu/127.8.102.60@KRBTEST.COM (short username kudu)
I20260926 15:04:46.643481 30975 webserver.cc:460] Webserver started at http://127.8.102.60:46761/ using document root <none> and password file <none>
I20260926 15:04:46.643764 30975 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:46.643843 30975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:46.644001 30975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:46.645108 30975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-2/data/instance:
uuid: "44855d7b42c041b5b91dda63f4df156d"
format_stamp: "Formatted at 2026-09-26 15:04:46 on dist-test-slave-jnhs"
server_key: "d58edd7b23c1566828926b42521afe75"
server_key_iv: "386c43a7154247a54ca85f5dca4f725f"
server_key_version: "encryptionkey@0"
I20260926 15:04:46.645234 30968 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } attempt: 2
I20260926 15:04:46.645550 30975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-2/wal/instance:
uuid: "44855d7b42c041b5b91dda63f4df156d"
format_stamp: "Formatted at 2026-09-26 15:04:46 on dist-test-slave-jnhs"
server_key: "d58edd7b23c1566828926b42521afe75"
server_key_iv: "386c43a7154247a54ca85f5dca4f725f"
server_key_version: "encryptionkey@0"
I20260926 15:04:46.645706 30970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.645540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:42155 (local address 127.0.0.1:44290)
0926 15:04:46.645603 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.645672 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260926 15:04:46.646072 30970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.645988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:42155 (local address 127.0.0.1:44298)
0926 15:04:46.646033 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.646057 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 15:04:46.646168 30968 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.60:42155: Network error: Client connection negotiation failed: client connection to 127.8.102.60:42155: connect: Connection refused (error 111)
I20260926 15:04:46.647323 30975 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.002s sys 0.002s
I20260926 15:04:46.648401 30990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:46.648581 30975 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:04:46.648680 30975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-2/wal
uuid: "44855d7b42c041b5b91dda63f4df156d"
format_stamp: "Formatted at 2026-09-26 15:04:46 on dist-test-slave-jnhs"
server_key: "d58edd7b23c1566828926b42521afe75"
server_key_iv: "386c43a7154247a54ca85f5dca4f725f"
server_key_version: "encryptionkey@0"
I20260926 15:04:46.648775 30975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:46.681074 30975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:46.683209 30893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } attempt: 2
I20260926 15:04:46.683629 30975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:46.683704 30897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.683560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:42155 (local address 127.0.0.1:44300)
0926 15:04:46.683626 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.683666 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260926 15:04:46.683792 30975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:46.684110 30897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.683984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:42155 (local address 127.0.0.1:44308)
0926 15:04:46.684037 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.684066 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:42155: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 15:04:46.684231 30893 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.60:42155: Network error: Client connection negotiation failed: client connection to 127.8.102.60:42155: connect: Connection refused (error 111)
I20260926 15:04:46.688114 30975 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.60:42155
I20260926 15:04:46.688239 31042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.60:42155 every 8 connection(s)
I20260926 15:04:46.688555 30975 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-2/data/info.pb
I20260926 15:04:46.689643 31043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:46.691376 31043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:04:46 dist-test-slave-jnhs krb5kdc[30812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435086, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.60@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:04:46.698563 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 30975
I20260926 15:04:46.698720 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790434810362722-8600-0/minicluster-data/master-2/wal/instance
I20260926 15:04:46.699034 8600 external_mini_cluster.cc:1428] Setting key ffa4f75109eb7c4202b841687830d45f
I20260926 15:04:46.701740 30896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.692103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47026 (local address 127.8.102.62:36159)
0926 15:04:46.692193 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.692195 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.692407 (+ 212us) server_negotiation.cc:407] Connection header received
0926 15:04:46.692862 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.692863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.692873 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.692949 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.693556 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.694049 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.695407 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.695594 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.695657 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.698019 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:46.698020 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:46.698020 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:46.698027 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:46.700000 (+ 1973us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.700611 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.700613 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.700613 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.700717 (+ 104us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.701000 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.701001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.701001 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.701154 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:46.701227 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:46.701540 (+ 313us) server_negotiation.cc:299] Negotiation successful
0926 15:04:46.701628 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260926 15:04:46.701849 31045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.692088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:36159 (local address 127.0.0.1:47026)
0926 15:04:46.692277 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.692320 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:46.692777 (+ 457us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:46.692998 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:46.693001 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:46.693252 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:46.693447 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:46.693451 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.694160 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.694161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:46.695297 (+ 1136us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:46.695300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.695707 (+ 407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.695708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:46.695825 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.696508 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:46.696517 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:46.697923 (+ 1406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:46.700089 (+ 2166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:46.700097 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:46.700099 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:46.700491 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:46.700806 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:46.700807 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:46.700807 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:46.700920 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:46.701260 (+ 340us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:46.701266 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:46.701397 (+ 131us) client_negotiation.cc:769] Sending connection context
0926 15:04:46.701497 (+ 100us) client_negotiation.cc:240] Negotiation successful
0926 15:04:46.701631 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":110,"threads_started":1}
I20260926 15:04:46.702536 31043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:04:46 dist-test-slave-jnhs krb5kdc[30812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.60@KRBTEST.COM
Sep 26 15:04:46 dist-test-slave-jnhs krb5kdc[30812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435086, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.60@KRBTEST.COM for kudu/127.8.102.61@KRBTEST.COM
I20260926 15:04:46.709066 31047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.699514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44322 (local address 127.8.102.60:42155)
0926 15:04:46.699761 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.699767 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.699800 (+ 33us) server_negotiation.cc:407] Connection header received
0926 15:04:46.699827 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.699829 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.699847 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.699941 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.700682 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.701674 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.702864 (+ 1190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.703306 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.703398 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.705873 (+ 2475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:46.705883 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:46.705884 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:46.705891 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:46.707653 (+ 1762us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.708059 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.708062 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.708063 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.708123 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.708387 (+ 264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.708388 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.708388 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.708589 (+ 201us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:46.708757 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:46.708835 (+ 78us) server_negotiation.cc:299] Negotiation successful
0926 15:04:46.708917 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":205,"thread_start_us":118,"threads_started":1}
I20260926 15:04:46.711109 31045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.702835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:34855 (local address 127.0.0.1:32986)
0926 15:04:46.702886 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.702900 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:46.702935 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:46.703112 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:46.703112 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:46.703233 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:46.703282 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:46.703283 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.704826 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.704827 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:46.705363 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:46.705367 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.705871 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.705872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:46.705952 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.706340 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:46.706344 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:46.707809 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:46.709828 (+ 2019us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:46.709829 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:46.709830 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:46.710112 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:46.710373 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:46.710373 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:46.710374 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:46.710435 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:46.710760 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:46.710761 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:46.710815 (+ 54us) client_negotiation.cc:769] Sending connection context
0926 15:04:46.710905 (+ 90us) client_negotiation.cc:240] Negotiation successful
0926 15:04:46.711016 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260926 15:04:46.711135 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.702920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32986 (local address 127.8.102.61:34855)
0926 15:04:46.702987 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.702989 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.702997 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:04:46.703021 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.703022 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.703032 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.703097 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.703359 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.703887 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.705458 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.705783 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.705855 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.707901 (+ 2046us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:46.707902 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:46.707902 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:46.707909 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:46.709765 (+ 1856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.710203 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.710205 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.710205 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.710276 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.710505 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.710506 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.710506 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.710615 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:46.710766 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:46.710966 (+ 200us) server_negotiation.cc:299] Negotiation successful
0926 15:04:46.711044 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260926 15:04:46.711593 31043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:04:46 dist-test-slave-jnhs krb5kdc[30812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435086, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.60@KRBTEST.COM for kudu/127.8.102.60@KRBTEST.COM
Sep 26 15:04:46 dist-test-slave-jnhs krb5kdc[30812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:04:46.719390 31045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.711910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:42155 (local address 127.0.0.1:44326)
0926 15:04:46.711996 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.712011 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:46.712051 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:46.712167 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:46.712167 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:46.712292 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:46.712331 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:46.712333 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.713065 (+ 732us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.713066 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:46.713728 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:46.713731 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.714119 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.714121 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:46.714168 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.714609 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:46.714613 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:46.715913 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:46.718089 (+ 2176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:46.718090 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:46.718090 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:46.718277 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:46.718525 (+ 248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:46.718526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:46.718526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:46.718697 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:46.719013 (+ 316us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:46.719014 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:46.719052 (+ 38us) client_negotiation.cc:769] Sending connection context
0926 15:04:46.719215 (+ 163us) client_negotiation.cc:240] Negotiation successful
0926 15:04:46.719318 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260926 15:04:46.719643 31047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.711958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44326 (local address 127.8.102.60:42155)
0926 15:04:46.711998 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.712000 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.712042 (+ 42us) server_negotiation.cc:407] Connection header received
0926 15:04:46.712090 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.712091 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.712101 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.712158 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.712410 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.712977 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.713797 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.714008 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.714099 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.716040 (+ 1941us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:46.716041 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:46.716041 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:46.716130 (+ 89us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:46.717996 (+ 1866us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.718359 (+ 363us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.718360 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.718361 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.718423 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.718789 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.718789 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.718789 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.718901 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:46.718993 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:46.719298 (+ 305us) server_negotiation.cc:299] Negotiation successful
0926 15:04:46.719381 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 15:04:46.720250 30896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.710148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47038 (local address 127.8.102.62:36159)
0926 15:04:46.710221 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.710231 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.710240 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:04:46.710264 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.710265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.710276 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.710339 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.711646 (+ 1307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.712435 (+ 789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.713237 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.713438 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.713496 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.716721 (+ 3225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:46.716730 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:46.716730 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:46.716739 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:46.718392 (+ 1653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.718965 (+ 573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.718966 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.718967 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.719025 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.719244 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.719245 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.719245 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.719388 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:46.720002 (+ 614us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:46.720068 (+ 66us) server_negotiation.cc:299] Negotiation successful
0926 15:04:46.720160 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260926 15:04:46.721448 31043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d: Bootstrap starting.
I20260926 15:04:46.722558 31043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:46.722903 31043 log.cc:824] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:46.723946 31043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d: No bootstrap required, opened a new log
I20260926 15:04:46.726357 31043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94f523df154457cb556cd8703eadbc0" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } } peers { permanent_uuid: "9f1e411dd090483aae8dd1d6a2570997" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } } peers { permanent_uuid: "44855d7b42c041b5b91dda63f4df156d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } }
I20260926 15:04:46.726554 31043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44855d7b42c041b5b91dda63f4df156d, State: Initialized, Role: FOLLOWER
I20260926 15:04:46.726642 31043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94f523df154457cb556cd8703eadbc0" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } } peers { permanent_uuid: "9f1e411dd090483aae8dd1d6a2570997" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } } peers { permanent_uuid: "44855d7b42c041b5b91dda63f4df156d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } }
I20260926 15:04:46.727034 31043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:46.727121 31049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94f523df154457cb556cd8703eadbc0" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } } peers { permanent_uuid: "9f1e411dd090483aae8dd1d6a2570997" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } } peers { permanent_uuid: "44855d7b42c041b5b91dda63f4df156d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } } }
I20260926 15:04:46.727222 31049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 15:04:46 dist-test-slave-jnhs krb5kdc[30812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435086, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.61@KRBTEST.COM
W20260926 15:04:46.730901 31060 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:04:46.730963 31060 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:04:46.730058 30975 master_runner.cc:186] Error getting master registration for 127.8.102.62:36159: OK, instance_id { permanent_uuid: "c94f523df154457cb556cd8703eadbc0" instance_seqno: 1790435086333280 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 15:04:46.732750 30975 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 15:04:46.734377 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.720949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32994 (local address 127.8.102.61:34855)
0926 15:04:46.721015 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.721018 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.721026 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:04:46.721050 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.721051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.721061 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.721122 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.725574 (+ 4452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.726316 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.727357 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.727637 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.727717 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.731099 (+ 3382us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:46.731101 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:46.731101 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:46.731111 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:46.733076 (+ 1965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.733455 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.733457 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.733457 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.733528 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.733790 (+ 262us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.733791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.733792 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.733945 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:46.734108 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:46.734180 (+ 72us) server_negotiation.cc:299] Negotiation successful
0926 15:04:46.734269 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260926 15:04:46.736311 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:04:46.737301 30968 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } attempt: 3
I20260926 15:04:46.737449 30867 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "c94f523df154457cb556cd8703eadbc0"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47038
Sep 26 15:04:46 dist-test-slave-jnhs krb5kdc[30812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435086, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.61@KRBTEST.COM for kudu/127.8.102.60@KRBTEST.COM
I20260926 15:04:46.747048 31047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.738744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44330 (local address 127.8.102.60:42155)
0926 15:04:46.738791 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.738793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.738824 (+ 31us) server_negotiation.cc:407] Connection header received
0926 15:04:46.738899 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.738900 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.738909 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.738953 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.739315 (+ 362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.740036 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.741186 (+ 1150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.741503 (+ 317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.741572 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.743971 (+ 2399us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:46.743973 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:46.743974 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:46.743982 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:46.745685 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.746083 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.746084 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.746085 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.746154 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.746394 (+ 240us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.746395 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.746396 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.746557 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:46.746641 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:46.746860 (+ 219us) server_negotiation.cc:299] Negotiation successful
0926 15:04:46.746946 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 15:04:46.747048 30970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.738711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:42155 (local address 127.0.0.1:44330)
0926 15:04:46.738772 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.738784 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:46.738838 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:46.738990 (+ 152us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:46.738991 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:46.739175 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:46.739231 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:46.739233 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.740134 (+ 901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.740135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:46.740967 (+ 832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:46.740969 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.741596 (+ 627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.741597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:46.741669 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.742255 (+ 586us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:46.742262 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:46.743881 (+ 1619us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:46.745807 (+ 1926us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:46.745809 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:46.745810 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:46.746000 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:46.746246 (+ 246us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:46.746247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:46.746248 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:46.746318 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:46.746680 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:46.746681 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:46.746741 (+ 60us) client_negotiation.cc:769] Sending connection context
0926 15:04:46.746834 (+ 93us) client_negotiation.cc:240] Negotiation successful
0926 15:04:46.746946 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 15:04:46.749047 30968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997: Bootstrap starting.
I20260926 15:04:46.749751 30968 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:46.750015 30968 log.cc:824] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:46.751024 30968 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997: No bootstrap required, opened a new log
I20260926 15:04:46.752949 30968 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94f523df154457cb556cd8703eadbc0" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } } peers { permanent_uuid: "9f1e411dd090483aae8dd1d6a2570997" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } } peers { permanent_uuid: "44855d7b42c041b5b91dda63f4df156d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } }
I20260926 15:04:46.753072 30968 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9f1e411dd090483aae8dd1d6a2570997, State: Initialized, Role: FOLLOWER
I20260926 15:04:46.753136 30968 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94f523df154457cb556cd8703eadbc0" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } } peers { permanent_uuid: "9f1e411dd090483aae8dd1d6a2570997" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } } peers { permanent_uuid: "44855d7b42c041b5b91dda63f4df156d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } }
I20260926 15:04:46.753472 31063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94f523df154457cb556cd8703eadbc0" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } } peers { permanent_uuid: "9f1e411dd090483aae8dd1d6a2570997" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } } peers { permanent_uuid: "44855d7b42c041b5b91dda63f4df156d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } } }
I20260926 15:04:46.753535 30968 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:46.753621 31063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997 [sys.catalog]: This master's current role is: FOLLOWER
W20260926 15:04:46.755586 31074 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:04:46.755714 31074 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:04:46.755788 30900 master_runner.cc:186] Error getting master registration for 127.8.102.62:36159: OK, instance_id { permanent_uuid: "c94f523df154457cb556cd8703eadbc0" instance_seqno: 1790435086333280 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 15:04:46.758844 30900 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 15:04:46.783403 30893 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } attempt: 3
Sep 26 15:04:46 dist-test-slave-jnhs krb5kdc[30812](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435086, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for kudu/127.8.102.60@KRBTEST.COM
I20260926 15:04:46.792531 30897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.783808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:42155 (local address 127.0.0.1:44340)
0926 15:04:46.783872 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:46.783884 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:46.783928 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:46.784116 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:46.784117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:46.784320 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:46.784378 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:46.784381 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.785353 (+ 972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.785355 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:46.786199 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:46.786203 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.786733 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.786735 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:46.786803 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.787339 (+ 536us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:46.787345 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:46.789018 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:46.791255 (+ 2237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:46.791256 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:46.791257 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:46.791519 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:46.791755 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:46.791755 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:46.791756 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:46.791819 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:46.792175 (+ 356us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:46.792177 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:46.792236 (+ 59us) client_negotiation.cc:769] Sending connection context
0926 15:04:46.792304 (+ 68us) client_negotiation.cc:240] Negotiation successful
0926 15:04:46.792419 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 15:04:46.792963 31047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.783887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.8.102.60:42155)
0926 15:04:46.783934 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.783936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.783950 (+ 14us) server_negotiation.cc:407] Connection header received
0926 15:04:46.783983 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.783984 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.783995 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.784076 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.784468 (+ 392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.785235 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.786297 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.786609 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.786724 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.789119 (+ 2395us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:46.789121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:46.789122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:46.789180 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:46.791138 (+ 1958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.791620 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.791621 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.791621 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.791693 (+ 72us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.791893 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.791893 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.791894 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.792053 (+ 159us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:46.792139 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:46.792775 (+ 636us) server_negotiation.cc:299] Negotiation successful
0926 15:04:46.792869 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 15:04:46.794955 30893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0: Bootstrap starting.
I20260926 15:04:46.795794 30893 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:46.796155 30893 log.cc:824] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:46.797137 30893 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0: No bootstrap required, opened a new log
I20260926 15:04:46.798619 30893 raft_consensus.cc:348] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94f523df154457cb556cd8703eadbc0" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } } peers { permanent_uuid: "9f1e411dd090483aae8dd1d6a2570997" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } } peers { permanent_uuid: "44855d7b42c041b5b91dda63f4df156d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } }
I20260926 15:04:46.798835 30893 raft_consensus.cc:729] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c94f523df154457cb556cd8703eadbc0, State: Initialized, Role: FOLLOWER
I20260926 15:04:46.798928 30893 consensus_queue.cc:260] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94f523df154457cb556cd8703eadbc0" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } } peers { permanent_uuid: "9f1e411dd090483aae8dd1d6a2570997" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } } peers { permanent_uuid: "44855d7b42c041b5b91dda63f4df156d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } }
I20260926 15:04:46.799326 30893 sys_catalog.cc:564] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:46.799350 30867 raft_consensus.cc:482] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 15:04:46.799401 30867 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:46.800259 30867 raft_consensus.cc:504] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94f523df154457cb556cd8703eadbc0" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } } peers { permanent_uuid: "9f1e411dd090483aae8dd1d6a2570997" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } } peers { permanent_uuid: "44855d7b42c041b5b91dda63f4df156d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } }
I20260926 15:04:46.800544 30867 leader_election.cc:288] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [CANDIDATE]: Term 1 election: Requested vote from peers 9f1e411dd090483aae8dd1d6a2570997 (127.8.102.61:34855), 44855d7b42c041b5b91dda63f4df156d (127.8.102.60:42155)
I20260926 15:04:46.801258 30942 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c94f523df154457cb556cd8703eadbc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9f1e411dd090483aae8dd1d6a2570997"
I20260926 15:04:46.801414 30942 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:46.802341 30942 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c94f523df154457cb556cd8703eadbc0 in term 1.
W20260926 15:04:46.802518 31088 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:04:46.802567 31088 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:04:46.803292 30843 leader_election.cc:302] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9f1e411dd090483aae8dd1d6a2570997, c94f523df154457cb556cd8703eadbc0; no voters:
I20260926 15:04:46.804569 31017 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c94f523df154457cb556cd8703eadbc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "44855d7b42c041b5b91dda63f4df156d"
I20260926 15:04:46.804697 31017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:46.805549 31017 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c94f523df154457cb556cd8703eadbc0 in term 1.
I20260926 15:04:46.806718 31090 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:46.806923 31090 raft_consensus.cc:686] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [term 1 LEADER]: Becoming Leader. State: Replica: c94f523df154457cb556cd8703eadbc0, State: Running, Role: LEADER
I20260926 15:04:46.807020 31090 consensus_queue.cc:237] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94f523df154457cb556cd8703eadbc0" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } } peers { permanent_uuid: "9f1e411dd090483aae8dd1d6a2570997" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } } peers { permanent_uuid: "44855d7b42c041b5b91dda63f4df156d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } }
I20260926 15:04:46.807663 30825 master_runner.cc:263] No leader master found from master c94f523df154457cb556cd8703eadbc0
I20260926 15:04:46.807710 30825 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 15:04:46.799286 31077 sys_catalog.cc:455] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94f523df154457cb556cd8703eadbc0" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } } peers { permanent_uuid: "9f1e411dd090483aae8dd1d6a2570997" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } } peers { permanent_uuid: "44855d7b42c041b5b91dda63f4df156d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } } }
I20260926 15:04:46.807929 31077 sys_catalog.cc:458] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:04:46.815342 31077 sys_catalog.cc:455] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c94f523df154457cb556cd8703eadbc0. Latest consensus state: current_term: 1 leader_uuid: "c94f523df154457cb556cd8703eadbc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94f523df154457cb556cd8703eadbc0" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } } peers { permanent_uuid: "9f1e411dd090483aae8dd1d6a2570997" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } } peers { permanent_uuid: "44855d7b42c041b5b91dda63f4df156d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } } }
I20260926 15:04:46.815477 31077 sys_catalog.cc:458] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:46.817911 31047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.808974 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.8.102.60:42155)
0926 15:04:46.809016 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.809017 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.809573 (+ 556us) server_negotiation.cc:407] Connection header received
0926 15:04:46.809681 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.809682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.809693 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.809736 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.810697 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.811440 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.812605 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.812910 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.812953 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.814681 (+ 1728us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:46.814683 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:46.814683 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:46.814692 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:46.816639 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.817064 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.817065 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.817065 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.817120 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.817323 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.817324 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.817324 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.817453 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:46.817523 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:46.817740 (+ 217us) server_negotiation.cc:299] Negotiation successful
0926 15:04:46.817821 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 15:04:46.822121 31100 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:46.822345 31100 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:46.827566 30896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.809338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47042 (local address 127.8.102.62:36159)
0926 15:04:46.809386 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.809388 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.817074 (+ 7686us) server_negotiation.cc:407] Connection header received
0926 15:04:46.817188 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.817189 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.817200 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.817231 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.818820 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.819564 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.820795 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.821063 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.821139 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.824474 (+ 3335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:46.824475 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:46.824476 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:46.824483 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:46.826199 (+ 1716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.826617 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.826618 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.826619 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.826700 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.826924 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.826925 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.826926 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.827062 (+ 136us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:46.827164 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:46.827384 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 15:04:46.827468 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260926 15:04:46.828171 30972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:46.808474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33010 (local address 127.8.102.61:34855)
0926 15:04:46.808519 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.808521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.809578 (+ 1057us) server_negotiation.cc:407] Connection header received
0926 15:04:46.809686 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.809687 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.809697 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.809728 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.815142 (+ 5414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.815872 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.817880 (+ 2008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:46.818200 (+ 320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:46.818242 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:46.821766 (+ 3524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:46.821768 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:46.821768 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:46.821776 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:46.823644 (+ 1868us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.827304 (+ 3660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.827306 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.827306 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.827369 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:46.827564 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:46.827565 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:46.827565 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:46.827692 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:46.827772 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:46.827959 (+ 187us) server_negotiation.cc:299] Negotiation successful
0926 15:04:46.828062 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 15:04:46.830860 30846 tablet.cc:2369] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260926 15:04:46.835197 31017 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d [term 1 FOLLOWER]: Refusing update from remote peer c94f523df154457cb556cd8703eadbc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:04:46.835563 30942 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997 [term 1 FOLLOWER]: Refusing update from remote peer c94f523df154457cb556cd8703eadbc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:04:46.836161 31095 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9f1e411dd090483aae8dd1d6a2570997" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:04:46.836678 31095 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "44855d7b42c041b5b91dda63f4df156d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:04:46.838176 31063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c94f523df154457cb556cd8703eadbc0. Latest consensus state: current_term: 1 leader_uuid: "c94f523df154457cb556cd8703eadbc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94f523df154457cb556cd8703eadbc0" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } } peers { permanent_uuid: "9f1e411dd090483aae8dd1d6a2570997" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } } peers { permanent_uuid: "44855d7b42c041b5b91dda63f4df156d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } } }
I20260926 15:04:46.838271 31063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:04:46.840370 31049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c94f523df154457cb556cd8703eadbc0. Latest consensus state: current_term: 1 leader_uuid: "c94f523df154457cb556cd8703eadbc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94f523df154457cb556cd8703eadbc0" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } } peers { permanent_uuid: "9f1e411dd090483aae8dd1d6a2570997" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } } peers { permanent_uuid: "44855d7b42c041b5b91dda63f4df156d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } } }
I20260926 15:04:46.840463 31049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:04:46.842046 31077 sys_catalog.cc:455] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c94f523df154457cb556cd8703eadbc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94f523df154457cb556cd8703eadbc0" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } } peers { permanent_uuid: "9f1e411dd090483aae8dd1d6a2570997" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } } peers { permanent_uuid: "44855d7b42c041b5b91dda63f4df156d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } } }
I20260926 15:04:46.842147 31077 sys_catalog.cc:458] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:46.842175 31049 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c94f523df154457cb556cd8703eadbc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94f523df154457cb556cd8703eadbc0" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } } peers { permanent_uuid: "9f1e411dd090483aae8dd1d6a2570997" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } } peers { permanent_uuid: "44855d7b42c041b5b91dda63f4df156d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } } }
I20260926 15:04:46.842243 31049 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:04:46.842211 31101 mvcc.cc:205] Tried to move back new op lower bound from 7333622115673206784 to 7333622115593129984. Current Snapshot: MvccSnapshot[applied={T|T < 7333622115673206784}]
I20260926 15:04:46.842523 31104 mvcc.cc:205] Tried to move back new op lower bound from 7333622115673206784 to 7333622115593129984. Current Snapshot: MvccSnapshot[applied={T|T < 7333622115673206784}]
I20260926 15:04:46.843536 31100 catalog_manager.cc:1347] Generated new cluster ID: 79f1243fff3e4af48c0c84b832a1e165
I20260926 15:04:46.843585 31100 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:46.843609 31090 sys_catalog.cc:455] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c94f523df154457cb556cd8703eadbc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94f523df154457cb556cd8703eadbc0" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } } peers { permanent_uuid: "9f1e411dd090483aae8dd1d6a2570997" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } } peers { permanent_uuid: "44855d7b42c041b5b91dda63f4df156d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } } }
I20260926 15:04:46.843693 31090 sys_catalog.cc:458] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:46.844235 31102 mvcc.cc:205] Tried to move back new op lower bound from 7333622115673206784 to 7333622115593129984. Current Snapshot: MvccSnapshot[applied={T|T < 7333622115673206784}]
I20260926 15:04:46.847041 31063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c94f523df154457cb556cd8703eadbc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c94f523df154457cb556cd8703eadbc0" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 36159 } } peers { permanent_uuid: "9f1e411dd090483aae8dd1d6a2570997" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 34855 } } peers { permanent_uuid: "44855d7b42c041b5b91dda63f4df156d" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 42155 } } }
I20260926 15:04:46.847121 31063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:04:46.862824 31100 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:46.863802 31100 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:46.878795 31100 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c94f523df154457cb556cd8703eadbc0: Generated new TSK 0
I20260926 15:04:46.879153 31100 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 15:04:46.905968 30972 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:46.904187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33020 (local address 127.8.102.61:34855)
0926 15:04:46.904234 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.904236 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.905282 (+ 1046us) server_negotiation.cc:407] Connection header received
0926 15:04:46.905400 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.905401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.905412 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.905442 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.905892 (+ 450us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33020: BlockingRecv error: recv got EOF from 127.0.0.1:33020 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 15:04:46.907073 31047 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:46.904605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44358 (local address 127.8.102.60:42155)
0926 15:04:46.904664 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.904667 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.906428 (+ 1761us) server_negotiation.cc:407] Connection header received
0926 15:04:46.906541 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.906543 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.906556 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.906587 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.907001 (+ 414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44358: BlockingRecv error: recv got EOF from 127.0.0.1:44358 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 15:04:46.907948 30896 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:46.905014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47052 (local address 127.8.102.62:36159)
0926 15:04:46.905063 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.905065 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.907438 (+ 2373us) server_negotiation.cc:407] Connection header received
0926 15:04:46.907532 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.907533 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.907543 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.907572 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.907889 (+ 317us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47052: BlockingRecv error: recv got EOF from 127.0.0.1:47052 (error 108)
Metrics: {"server-negotiator.queue_time_us":25}
W20260926 15:04:46.910558 30972 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:46.909339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33034 (local address 127.8.102.61:34855)
0926 15:04:46.909374 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.909375 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.910059 (+ 684us) server_negotiation.cc:407] Connection header received
0926 15:04:46.910162 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.910163 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.910181 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.910209 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.910510 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33034: BlockingRecv error: recv got EOF from 127.0.0.1:33034 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260926 15:04:46.913604 30972 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:46.912859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33042 (local address 127.8.102.61:34855)
0926 15:04:46.912892 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.912894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.913124 (+ 230us) server_negotiation.cc:407] Connection header received
0926 15:04:46.913216 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.913217 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.913227 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.913255 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.913554 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33042: BlockingRecv error: recv got EOF from 127.0.0.1:33042 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 15:04:46.917673 30972 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:46.916845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33058 (local address 127.8.102.61:34855)
0926 15:04:46.916884 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.916885 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.917130 (+ 245us) server_negotiation.cc:407] Connection header received
0926 15:04:46.917228 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.917229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.917240 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.917267 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.917609 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33058: BlockingRecv error: recv got EOF from 127.0.0.1:33058 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260926 15:04:46.923859 30972 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:46.923130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33068 (local address 127.8.102.61:34855)
0926 15:04:46.923171 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.923174 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.923338 (+ 164us) server_negotiation.cc:407] Connection header received
0926 15:04:46.923408 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.923409 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.923421 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.923472 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.923803 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33068: BlockingRecv error: recv got EOF from 127.0.0.1:33068 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 15:04:46.933672 30972 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:46.932892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33084 (local address 127.8.102.61:34855)
0926 15:04:46.932944 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.932946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.933068 (+ 122us) server_negotiation.cc:407] Connection header received
0926 15:04:46.933153 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.933154 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.933165 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.933221 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.933596 (+ 375us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33084: BlockingRecv error: recv got EOF from 127.0.0.1:33084 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 15:04:46.951754 30972 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:46.950924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33090 (local address 127.8.102.61:34855)
0926 15:04:46.950972 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.950975 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.951124 (+ 149us) server_negotiation.cc:407] Connection header received
0926 15:04:46.951207 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.951208 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.951221 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.951298 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.951668 (+ 370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33090: BlockingRecv error: recv got EOF from 127.0.0.1:33090 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 15:04:46.985668 30972 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:46.984894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33094 (local address 127.8.102.61:34855)
0926 15:04:46.984968 (+ 74us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:46.984970 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:46.985061 (+ 91us) server_negotiation.cc:407] Connection header received
0926 15:04:46.985099 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:46.985100 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:46.985113 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:46.985209 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:46.985602 (+ 393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33094: BlockingRecv error: recv got EOF from 127.0.0.1:33094 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 15:04:47.051931 30972 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:47.051044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33096 (local address 127.8.102.61:34855)
0926 15:04:47.051113 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:47.051115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:47.051129 (+ 14us) server_negotiation.cc:407] Connection header received
0926 15:04:47.051270 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:47.051271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:47.051283 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:47.051345 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:47.051832 (+ 487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33096: BlockingRecv error: recv got EOF from 127.0.0.1:33096 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
W20260926 15:04:47.181794 30972 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:47.181090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33100 (local address 127.8.102.61:34855)
0926 15:04:47.181192 (+ 102us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:47.181194 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:47.181237 (+ 43us) server_negotiation.cc:407] Connection header received
0926 15:04:47.181318 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:47.181318 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:47.181330 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:47.181383 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:47.181724 (+ 341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33100: BlockingRecv error: recv got EOF from 127.0.0.1:33100 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
W20260926 15:04:47.439931 30972 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:47.439192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33108 (local address 127.8.102.61:34855)
0926 15:04:47.439277 (+ 85us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:47.439279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:47.439333 (+ 54us) server_negotiation.cc:407] Connection header received
0926 15:04:47.439385 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:47.439386 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:47.439397 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:47.439455 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:47.439855 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33108: BlockingRecv error: recv got EOF from 127.0.0.1:33108 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260926 15:04:47.731442 31060 catalog_manager.cc:1259] Loaded cluster ID: 79f1243fff3e4af48c0c84b832a1e165
I20260926 15:04:47.731529 31060 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d: loading cluster ID for follower catalog manager: success
I20260926 15:04:47.732386 31060 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d: acquiring CA information for follower catalog manager: success
I20260926 15:04:47.732717 31060 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 44855d7b42c041b5b91dda63f4df156d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 15:04:47.756238 31074 catalog_manager.cc:1259] Loaded cluster ID: 79f1243fff3e4af48c0c84b832a1e165
I20260926 15:04:47.756326 31074 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997: loading cluster ID for follower catalog manager: success
I20260926 15:04:47.757118 31074 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997: acquiring CA information for follower catalog manager: success
I20260926 15:04:47.757376 31074 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9f1e411dd090483aae8dd1d6a2570997: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 15:04:47.956099 31173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:47.953169 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33118 (local address 127.8.102.61:34855)
0926 15:04:47.953307 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:47.953310 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:47.953323 (+ 13us) server_negotiation.cc:407] Connection header received
0926 15:04:47.953360 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:47.953361 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:47.953425 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:47.953502 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:04:47.953733 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:47.954275 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:47.955290 (+ 1015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:47.955499 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:47.955565 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:47.955717 (+ 152us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:04:47.955830 (+ 113us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:04:47.955891 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:47.955931 (+ 40us) server_negotiation.cc:299] Negotiation successful
0926 15:04:47.955951 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":46,"threads_started":1}
I20260926 15:04:47.960729 31179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:47.958110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44374 (local address 127.8.102.60:42155)
0926 15:04:47.958208 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:47.958210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:47.958223 (+ 13us) server_negotiation.cc:407] Connection header received
0926 15:04:47.958289 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:47.958289 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:47.958323 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:47.958394 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:04:47.958518 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:47.959061 (+ 543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:47.959863 (+ 802us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:47.960060 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:47.960128 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:47.960378 (+ 250us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:04:47.960483 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:04:47.960547 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:47.960645 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 15:04:47.960665 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":48,"threads_started":1}
I20260926 15:04:47.965394 31173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:47.962733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33132 (local address 127.8.102.61:34855)
0926 15:04:47.962775 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:47.962777 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:47.962784 (+ 7us) server_negotiation.cc:407] Connection header received
0926 15:04:47.962810 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:47.962810 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:47.962820 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:47.962891 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:04:47.963033 (+ 142us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:47.963640 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:47.964603 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:47.964909 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:47.964988 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:47.965128 (+ 140us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:04:47.965162 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:04:47.965211 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:47.965296 (+ 85us) server_negotiation.cc:299] Negotiation successful
0926 15:04:47.965317 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 15:04:47.966038 31187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:47.962651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47056 (local address 127.8.102.62:36159)
0926 15:04:47.962754 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:47.962756 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:47.962763 (+ 7us) server_negotiation.cc:407] Connection header received
0926 15:04:47.962813 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:47.962814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:47.962846 (+ 32us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:47.962947 (+ 101us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:04:47.963194 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:47.963979 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:47.964918 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:47.965251 (+ 333us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:47.965319 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:47.965577 (+ 258us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:04:47.965701 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:04:47.965758 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:47.965926 (+ 168us) server_negotiation.cc:299] Negotiation successful
0926 15:04:47.965951 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":51,"threads_started":1}
I20260926 15:04:47.966097 31179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:47.962676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44386 (local address 127.8.102.60:42155)
0926 15:04:47.962727 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:47.962730 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:47.963011 (+ 281us) server_negotiation.cc:407] Connection header received
0926 15:04:47.963082 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:47.963083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:47.963094 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:47.963145 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:04:47.963300 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:47.964010 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:47.965135 (+ 1125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:47.965417 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:47.965614 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:47.965785 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:04:47.965821 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:04:47.965865 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:47.966003 (+ 138us) server_negotiation.cc:299] Negotiation successful
0926 15:04:47.966024 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260926 15:04:47.967973 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 30825
I20260926 15:04:47.973353 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 30900
I20260926 15:04:47.979058 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 30975
2026-09-26T15:04:47Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2951 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 26 15:04:48 dist-test-slave-jnhs krb5kdc[31190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 26 15:04:48 dist-test-slave-jnhs krb5kdc[31190](info): set up 2 sockets
Sep 26 15:04:48 dist-test-slave-jnhs krb5kdc[31190](info): commencing operation
krb5kdc: starting...
W20260926 15:04:49.015149 8600 mini_kdc.cc:121] Time spent starting KDC: real 1.016s user 0.002s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/test-user.keytab.
Sep 26 15:04:49 dist-test-slave-jnhs krb5kdc[31190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-26T15:04:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-26T15:04:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.8.102.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
Entry for principal kudu/127.8.102.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab.
I20260926 15:04:49.142366 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:44829
--webserver_interface=127.8.102.62
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:44829,127.8.102.61:36487,127.8.102.60:34327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260926 15:04:49.225034 31203 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 15:04:49.225338 31203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:49.225369 31203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:49.225404 31203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:49.226614 31203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:49.226719 31203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:49.226745 31203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:49.226763 31203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:49.226783 31203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:49.228425 31203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:44829,127.8.102.61:36487,127.8.102.60:34327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.62.keytab
--principal=kudu/127.8.102.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.62:44829
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.8.102.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:49.228732 31203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:49.228976 31203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:49.233952 31211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:49.234027 31203 server_base.cc:1034] running on GCE node
W20260926 15:04:49.233908 31209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:49.233908 31208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:49.234601 31203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:49.234962 31203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:49.236166 31203 hybrid_clock.cc:648] HybridClock initialized: now 1790435089236112 us; error 65 us; skew 500 ppm
Sep 26 15:04:49 dist-test-slave-jnhs krb5kdc[31190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435089, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:49.239049 31203 init.cc:377] Logged in from keytab as kudu/127.8.102.62@KRBTEST.COM (short username kudu)
I20260926 15:04:49.239591 31203 webserver.cc:460] Webserver started at http://127.8.102.62:44537/ using document root <none> and password file <none>
I20260926 15:04:49.239841 31203 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:49.239889 31203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:49.240041 31203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:49.241154 31203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-0/data/instance:
uuid: "6fae2669468b45639a639e4fcf18b844"
format_stamp: "Formatted at 2026-09-26 15:04:49 on dist-test-slave-jnhs"
server_key: "acf20f0d30ae32a55e92e77f0baa7cda"
server_key_iv: "9fbaa7078b57e644b714e509a59214e1"
server_key_version: "encryptionkey@0"
I20260926 15:04:49.241528 31203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-0/wal/instance:
uuid: "6fae2669468b45639a639e4fcf18b844"
format_stamp: "Formatted at 2026-09-26 15:04:49 on dist-test-slave-jnhs"
server_key: "acf20f0d30ae32a55e92e77f0baa7cda"
server_key_iv: "9fbaa7078b57e644b714e509a59214e1"
server_key_version: "encryptionkey@0"
I20260926 15:04:49.243285 31203 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260926 15:04:49.244218 31218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:49.244529 31203 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:49.244616 31203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-0/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-0/wal
uuid: "6fae2669468b45639a639e4fcf18b844"
format_stamp: "Formatted at 2026-09-26 15:04:49 on dist-test-slave-jnhs"
server_key: "acf20f0d30ae32a55e92e77f0baa7cda"
server_key_iv: "9fbaa7078b57e644b714e509a59214e1"
server_key_version: "encryptionkey@0"
I20260926 15:04:49.244725 31203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:49.262547 31203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:49.265582 31203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:49.265713 31203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:49.270141 31203 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.62:44829
I20260926 15:04:49.270244 31270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.62:44829 every 8 connection(s)
I20260926 15:04:49.270720 31203 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-0/data/info.pb
I20260926 15:04:49.271937 31271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:49.273556 31271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:04:49.277887 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 31203
I20260926 15:04:49.278002 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-0/wal/instance
I20260926 15:04:49.278204 8600 external_mini_cluster.cc:1428] Setting key 86d825271a84188f74b8cd55218056f0
Sep 26 15:04:49 dist-test-slave-jnhs krb5kdc[31190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435089, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:04:49.284752 31273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.274280 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44829 (local address 127.0.0.1:59620)
0926 15:04:49.274426 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.274476 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:49.275026 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:49.275375 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:49.275378 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:49.275652 (+ 274us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:49.275816 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:49.275820 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.276985 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.276986 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:49.277686 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:49.277688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.278453 (+ 765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.278454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:49.278527 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.279216 (+ 689us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:49.279226 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:49.280938 (+ 1712us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:49.283176 (+ 2238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:49.283182 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:49.283185 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:49.283388 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:49.283663 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:49.283663 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:49.283664 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:49.283753 (+ 89us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:49.284133 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:49.284137 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:49.284209 (+ 72us) client_negotiation.cc:769] Sending connection context
0926 15:04:49.284322 (+ 113us) client_negotiation.cc:240] Negotiation successful
0926 15:04:49.284556 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":87,"thread_start_us":70,"threads_started":1}
I20260926 15:04:49.284750 31274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.274337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59620 (local address 127.8.102.62:44829)
0926 15:04:49.274538 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.274542 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.274574 (+ 32us) server_negotiation.cc:407] Connection header received
0926 15:04:49.275201 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.275204 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.275223 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.275343 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.275915 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.276861 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.277794 (+ 933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.278331 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.278462 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.281035 (+ 2573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:49.281043 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:49.281044 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:49.281052 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:49.283058 (+ 2006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.283495 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.283499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.283499 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.283581 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.283824 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.283825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.283826 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.284048 (+ 222us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:49.284139 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:49.284354 (+ 215us) server_negotiation.cc:299] Negotiation successful
0926 15:04:49.284495 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":173,"thread_start_us":95,"threads_started":1}
I20260926 15:04:49.285740 31271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:04:49.286149 31273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.285982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:36487 (local address 127.0.0.1:46192)
0926 15:04:49.286049 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.286108 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.61:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20260926 15:04:49.286337 31219 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.102.61:36487: connect: Connection refused (error 111)
I20260926 15:04:49.286914 31273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.286797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:36487 (local address 127.0.0.1:46194)
0926 15:04:49.286850 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.286885 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.61:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
WARNING: no policy specified for kudu/127.8.102.61@KRBTEST.COM; defaulting to no policy
W20260926 15:04:49.287067 31271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.61:36487: Network error: Client connection negotiation failed: client connection to 127.8.102.61:36487: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.61.keytab.
Entry for principal kudu/127.8.102.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.61.keytab.
I20260926 15:04:49.311271 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.61:36487
--webserver_interface=127.8.102.61
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.61.keytab
--principal=kudu/127.8.102.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:44829,127.8.102.61:36487,127.8.102.60:34327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:04:49.336302 31271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } attempt: 1
I20260926 15:04:49.337069 31273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.336795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:36487 (local address 127.0.0.1:46204)
0926 15:04:49.336980 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.337030 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.61:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":153}
I20260926 15:04:49.337503 31273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.337413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:36487 (local address 127.0.0.1:46210)
0926 15:04:49.337449 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.337480 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.61:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":21}
W20260926 15:04:49.337702 31271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.61:36487: Network error: Client connection negotiation failed: client connection to 127.8.102.61:36487: connect: Connection refused (error 111)
W20260926 15:04:49.397328 31278 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 15:04:49.397563 31278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:49.397583 31278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:49.397603 31278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:49.398892 31278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:49.398936 31278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:49.398950 31278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:49.398963 31278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:49.398976 31278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:49.400456 31278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:44829,127.8.102.61:36487,127.8.102.60:34327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.61.keytab
--principal=kudu/127.8.102.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.61:36487
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.8.102.61
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31278
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:49.400660 31278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:49.400847 31278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:49.405732 31284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:49.405831 31271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } attempt: 2
W20260926 15:04:49.405795 31286 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:49.405990 31278 server_base.cc:1034] running on GCE node
I20260926 15:04:49.406224 31273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.406080 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:36487 (local address 127.0.0.1:46218)
0926 15:04:49.406158 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.406183 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.61:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20260926 15:04:49.406313 31283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:49.406608 31273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.406482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:36487 (local address 127.0.0.1:46232)
0926 15:04:49.406544 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.406577 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.61:36487: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 15:04:49.406641 31278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260926 15:04:49.406765 31271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.61:36487: Network error: Client connection negotiation failed: client connection to 127.8.102.61:36487: connect: Connection refused (error 111)
I20260926 15:04:49.406980 31278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:49.408133 31278 hybrid_clock.cc:648] HybridClock initialized: now 1790435089408117 us; error 29 us; skew 500 ppm
Sep 26 15:04:49 dist-test-slave-jnhs krb5kdc[31190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435089, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:49.410706 31278 init.cc:377] Logged in from keytab as kudu/127.8.102.61@KRBTEST.COM (short username kudu)
I20260926 15:04:49.411187 31278 webserver.cc:460] Webserver started at http://127.8.102.61:37831/ using document root <none> and password file <none>
I20260926 15:04:49.411446 31278 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:49.411518 31278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:49.411669 31278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:49.412765 31278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-1/data/instance:
uuid: "7efd25aab3c549a89ef6360d2f68b81e"
format_stamp: "Formatted at 2026-09-26 15:04:49 on dist-test-slave-jnhs"
server_key: "04acb8c84e70f5206e7321e334eeb248"
server_key_iv: "504739a7ef43146b9bea6d2c29bd2fe7"
server_key_version: "encryptionkey@0"
I20260926 15:04:49.413128 31278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-1/wal/instance:
uuid: "7efd25aab3c549a89ef6360d2f68b81e"
format_stamp: "Formatted at 2026-09-26 15:04:49 on dist-test-slave-jnhs"
server_key: "04acb8c84e70f5206e7321e334eeb248"
server_key_iv: "504739a7ef43146b9bea6d2c29bd2fe7"
server_key_version: "encryptionkey@0"
I20260926 15:04:49.414461 31278 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.001s
I20260926 15:04:49.415387 31293 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:49.415578 31278 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260926 15:04:49.415684 31278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-1/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-1/wal
uuid: "7efd25aab3c549a89ef6360d2f68b81e"
format_stamp: "Formatted at 2026-09-26 15:04:49 on dist-test-slave-jnhs"
server_key: "04acb8c84e70f5206e7321e334eeb248"
server_key_iv: "504739a7ef43146b9bea6d2c29bd2fe7"
server_key_version: "encryptionkey@0"
I20260926 15:04:49.415772 31278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:49.434736 31278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:49.436839 31278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:49.437021 31278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:49.441571 31278 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.61:36487
I20260926 15:04:49.441612 31345 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.61:36487 every 8 connection(s)
I20260926 15:04:49.442018 31278 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-1/data/info.pb
I20260926 15:04:49.443039 31346 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:49.444603 31346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:04:49.446919 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 31278
I20260926 15:04:49.447060 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-1/wal/instance
I20260926 15:04:49.447266 8600 external_mini_cluster.cc:1428] Setting key 2e8692e2645adf0a44590bc91ec49862
Sep 26 15:04:49 dist-test-slave-jnhs krb5kdc[31190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435089, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.61@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:04:49.454509 31274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.445294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59630 (local address 127.8.102.62:44829)
0926 15:04:49.445340 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.445342 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.445559 (+ 217us) server_negotiation.cc:407] Connection header received
0926 15:04:49.445989 (+ 430us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.445990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.446002 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.446054 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.446674 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.447252 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.448374 (+ 1122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.448720 (+ 346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.448796 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.451380 (+ 2584us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:49.451382 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:49.451382 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:49.451390 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:49.452946 (+ 1556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.453459 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.453461 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.453461 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.453526 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.453829 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.453835 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.453835 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.453957 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:49.454032 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:49.454324 (+ 292us) server_negotiation.cc:299] Negotiation successful
0926 15:04:49.454414 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 15:04:49.454746 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.445279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44829 (local address 127.0.0.1:59630)
0926 15:04:49.445433 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.445468 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:49.445907 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:49.446105 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:49.446108 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:49.446357 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:49.446549 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:49.446552 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.447438 (+ 886us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.447439 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:49.448250 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:49.448253 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.448829 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.448830 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:49.448946 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.449571 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:49.449581 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:49.451254 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:49.453065 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:49.453073 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:49.453076 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:49.453358 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:49.453599 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:49.453600 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:49.453601 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:49.453734 (+ 133us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:49.454034 (+ 300us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:49.454042 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:49.454189 (+ 147us) client_negotiation.cc:769] Sending connection context
0926 15:04:49.454305 (+ 116us) client_negotiation.cc:240] Negotiation successful
0926 15:04:49.454467 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":84,"threads_started":1}
WARNING: no policy specified for kudu/127.8.102.60@KRBTEST.COM; defaulting to no policy
I20260926 15:04:49.455348 31346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.8.102.60@KRBTEST.COM" created.
Sep 26 15:04:49 dist-test-slave-jnhs krb5kdc[31190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435089, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.61@KRBTEST.COM for kudu/127.8.102.61@KRBTEST.COM
I20260926 15:04:49.463337 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.455618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:36487 (local address 127.0.0.1:46236)
0926 15:04:49.455682 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.455692 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:49.455738 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:49.456039 (+ 301us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:49.456040 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:49.456200 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:49.456311 (+ 111us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:49.456314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.457143 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.457144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:49.457762 (+ 618us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:49.457765 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.458263 (+ 498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.458263 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:49.458359 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.458899 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:49.458905 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:49.460316 (+ 1411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:49.462127 (+ 1811us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:49.462127 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:49.462128 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:49.462338 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:49.462580 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:49.462581 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:49.462581 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:49.462677 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:49.462965 (+ 288us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:49.462966 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:49.463006 (+ 40us) client_negotiation.cc:769] Sending connection context
0926 15:04:49.463077 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 15:04:49.463186 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260926 15:04:49.463353 31350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.455718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46236 (local address 127.8.102.61:36487)
0926 15:04:49.455899 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.455903 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.455919 (+ 16us) server_negotiation.cc:407] Connection header received
0926 15:04:49.455937 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.455938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.455949 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.456040 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.456399 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.457068 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.457847 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.458162 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.458219 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.460410 (+ 2191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:49.460420 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:49.460420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:49.460427 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:49.462019 (+ 1592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.462442 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.462445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.462445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.462502 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.462734 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.462735 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.462736 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.462889 (+ 153us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:49.462988 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:49.463116 (+ 128us) server_negotiation.cc:299] Negotiation successful
0926 15:04:49.463239 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":58,"server-negotiator.queue_time_us":156,"thread_start_us":97,"threads_started":1}
I20260926 15:04:49.464000 31346 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:04:49.464403 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.464282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:34327 (local address 127.0.0.1:46216)
0926 15:04:49.464338 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.464381 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260926 15:04:49.464576 31294 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.8.102.60:34327: connect: Connection refused (error 111)
I20260926 15:04:49.464977 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.464908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:34327 (local address 127.0.0.1:46226)
0926 15:04:49.464934 (+ 26us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.464959 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260926 15:04:49.465116 31346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.60:34327: Network error: Client connection negotiation failed: client connection to 127.8.102.60:34327: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.8.102.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.60.keytab.
Entry for principal kudu/127.8.102.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.60.keytab.
I20260926 15:04:49.478559 8600 external_mini_cluster.cc:1326] Running /tmp/dist-test-task17_aQt/build/release/bin/kudu
/tmp/dist-test-task17_aQt/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.60:34327
--webserver_interface=127.8.102.60
--webserver_port=0
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.60.keytab
--principal=kudu/127.8.102.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.8.102.20:46375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.8.102.62:44829,127.8.102.61:36487,127.8.102.60:34327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260926 15:04:49.497947 31271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } attempt: 3
Sep 26 15:04:49 dist-test-slave-jnhs krb5kdc[31190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435089, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for kudu/127.8.102.61@KRBTEST.COM
I20260926 15:04:49.505491 31350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.498525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46248 (local address 127.8.102.61:36487)
0926 15:04:49.498604 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.498606 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.498618 (+ 12us) server_negotiation.cc:407] Connection header received
0926 15:04:49.498668 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.498669 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.498682 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.498746 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.499081 (+ 335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.499580 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.500352 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.500572 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.500654 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.502790 (+ 2136us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:49.502791 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:49.502792 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:49.502799 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:49.504253 (+ 1454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.504634 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.504636 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.504636 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.504710 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.504935 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.504936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.504936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.505056 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:49.505137 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:49.505302 (+ 165us) server_negotiation.cc:299] Negotiation successful
0926 15:04:49.505386 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260926 15:04:49.505491 31273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.498421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:36487 (local address 127.0.0.1:46248)
0926 15:04:49.498493 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.498508 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:49.498593 (+ 85us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:49.498751 (+ 158us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:49.498752 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:49.498900 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:49.498984 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:49.498986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.499708 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.499709 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:49.500241 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:49.500243 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.500653 (+ 410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.500654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:49.500744 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.501241 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:49.501251 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:49.502692 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:49.504362 (+ 1670us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:49.504363 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:49.504364 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:49.504542 (+ 178us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:49.504783 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:49.504785 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:49.504785 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:49.504856 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:49.505134 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:49.505135 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:49.505186 (+ 51us) client_negotiation.cc:769] Sending connection context
0926 15:04:49.505277 (+ 91us) client_negotiation.cc:240] Negotiation successful
0926 15:04:49.505385 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 15:04:49.506156 31271 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:04:49.506507 31273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.506406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:34327 (local address 127.0.0.1:46238)
0926 15:04:49.506457 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.506487 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20260926 15:04:49.506867 31273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.506771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:34327 (local address 127.0.0.1:46254)
0926 15:04:49.506819 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.506843 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260926 15:04:49.506953 31271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.60:34327: Network error: Client connection negotiation failed: client connection to 127.8.102.60:34327: connect: Connection refused (error 111)
I20260926 15:04:49.514384 31346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } attempt: 1
I20260926 15:04:49.514925 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.514773 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:34327 (local address 127.0.0.1:46268)
0926 15:04:49.514841 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.514886 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
I20260926 15:04:49.515313 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.515214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:34327 (local address 127.0.0.1:46280)
0926 15:04:49.515263 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.515291 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260926 15:04:49.515441 31346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.60:34327: Network error: Client connection negotiation failed: client connection to 127.8.102.60:34327: connect: Connection refused (error 111)
I20260926 15:04:49.538120 31271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } attempt: 1
I20260926 15:04:49.538707 31273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.538488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:34327 (local address 127.0.0.1:46292)
0926 15:04:49.538560 (+ 72us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.538605 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260926 15:04:49.539081 31273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.538974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:34327 (local address 127.0.0.1:46294)
0926 15:04:49.539025 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.539050 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20260926 15:04:49.539177 31271 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.60:34327: Network error: Client connection negotiation failed: client connection to 127.8.102.60:34327: connect: Connection refused (error 111)
W20260926 15:04:49.563140 31353 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260926 15:04:49.563393 31353 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260926 15:04:49.563426 31353 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260926 15:04:49.563470 31353 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260926 15:04:49.564793 31353 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260926 15:04:49.564898 31353 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260926 15:04:49.564926 31353 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260926 15:04:49.564954 31353 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260926 15:04:49.564986 31353 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260926 15:04:49.566707 31353 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.8.102.20:46375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.8.102.62:44829,127.8.102.61:36487,127.8.102.60:34327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/krb5kdc/kudu_127.8.102.60.keytab
--principal=kudu/127.8.102.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.8.102.60:34327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.8.102.60
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31353
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 26 Sep 2026 13:59:08 UTC on ccycloud.quasar-qegwso.root.comops.site
build id 4674340
I20260926 15:04:49.567018 31353 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260926 15:04:49.567281 31353 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260926 15:04:49.572570 31361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:49.572597 31358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260926 15:04:49.572570 31359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260926 15:04:49.572682 31353 server_base.cc:1034] running on GCE node
I20260926 15:04:49.572968 31353 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260926 15:04:49.573287 31353 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260926 15:04:49.574433 31353 hybrid_clock.cc:648] HybridClock initialized: now 1790435089574411 us; error 40 us; skew 500 ppm
Sep 26 15:04:49 dist-test-slave-jnhs krb5kdc[31190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435089, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260926 15:04:49.577142 31353 init.cc:377] Logged in from keytab as kudu/127.8.102.60@KRBTEST.COM (short username kudu)
I20260926 15:04:49.577551 31353 webserver.cc:460] Webserver started at http://127.8.102.60:45305/ using document root <none> and password file <none>
I20260926 15:04:49.577831 31353 fs_manager.cc:359] Metadata directory not provided
I20260926 15:04:49.577880 31353 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260926 15:04:49.578020 31353 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260926 15:04:49.579228 31353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-2/data/instance:
uuid: "dcf4f79405a24893973978aacc782eb0"
format_stamp: "Formatted at 2026-09-26 15:04:49 on dist-test-slave-jnhs"
server_key: "973bed5f2df0abd61ac32be3f8260f4f"
server_key_iv: "e2c2ab675540a71eddffe5e973417eed"
server_key_version: "encryptionkey@0"
I20260926 15:04:49.579631 31353 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-2/wal/instance:
uuid: "dcf4f79405a24893973978aacc782eb0"
format_stamp: "Formatted at 2026-09-26 15:04:49 on dist-test-slave-jnhs"
server_key: "973bed5f2df0abd61ac32be3f8260f4f"
server_key_iv: "e2c2ab675540a71eddffe5e973417eed"
server_key_version: "encryptionkey@0"
I20260926 15:04:49.581020 31353 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260926 15:04:49.581851 31368 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260926 15:04:49.582099 31353 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260926 15:04:49.582167 31353 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-2/data,/tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-2/wal
uuid: "dcf4f79405a24893973978aacc782eb0"
format_stamp: "Formatted at 2026-09-26 15:04:49 on dist-test-slave-jnhs"
server_key: "973bed5f2df0abd61ac32be3f8260f4f"
server_key_iv: "e2c2ab675540a71eddffe5e973417eed"
server_key_version: "encryptionkey@0"
I20260926 15:04:49.582216 31353 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260926 15:04:49.583613 31346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } attempt: 2
I20260926 15:04:49.584026 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.583897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:34327 (local address 127.0.0.1:46308)
0926 15:04:49.583962 (+ 65us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.583999 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260926 15:04:49.584400 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.584304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:34327 (local address 127.0.0.1:46312)
0926 15:04:49.584351 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.584376 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.8.102.60:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20260926 15:04:49.584498 31346 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.8.102.60:34327: Network error: Client connection negotiation failed: client connection to 127.8.102.60:34327: connect: Connection refused (error 111)
I20260926 15:04:49.597071 31353 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260926 15:04:49.599170 31353 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260926 15:04:49.599350 31353 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260926 15:04:49.603307 31353 rpc_server.cc:307] RPC server started. Bound to: 127.8.102.60:34327
I20260926 15:04:49.603426 31420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.8.102.60:34327 every 8 connection(s)
I20260926 15:04:49.603749 31353 server_base.cc:1166] Dumped server information to /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-2/data/info.pb
I20260926 15:04:49.604950 31421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260926 15:04:49.606562 31421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:04:49 dist-test-slave-jnhs krb5kdc[31190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435089, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.60@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:04:49.613879 8600 external_mini_cluster.cc:1388] Started /tmp/dist-test-task17_aQt/build/release/bin/kudu as pid 31353
I20260926 15:04:49.613983 8600 external_mini_cluster.cc:1402] Reading /tmp/dist-test-task17_aQt/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790434810362722-8600-0/minicluster-data/master-2/wal/instance
I20260926 15:04:49.614307 8600 external_mini_cluster.cc:1428] Setting key bd11c77507da81fc30e901c9d20c2565
I20260926 15:04:49.614324 31271 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } attempt: 2
I20260926 15:04:49.616148 31274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.607409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59640 (local address 127.8.102.62:44829)
0926 15:04:49.607507 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.607510 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.607692 (+ 182us) server_negotiation.cc:407] Connection header received
0926 15:04:49.608182 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.608183 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.608194 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.608247 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.608824 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.609282 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.610327 (+ 1045us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.610517 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.610578 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.612907 (+ 2329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:49.612908 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:49.612909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:49.612916 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:49.614493 (+ 1577us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.615076 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.615077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.615078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.615151 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.615464 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.615465 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.615466 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.615584 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:49.615660 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:49.615959 (+ 299us) server_negotiation.cc:299] Negotiation successful
0926 15:04:49.616039 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260926 15:04:49.616370 31423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.607343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.62:44829 (local address 127.0.0.1:59640)
0926 15:04:49.607556 (+ 213us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.607594 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:49.608061 (+ 467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:49.608300 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:49.608303 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:49.608555 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:49.608749 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:49.608753 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.609365 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.609366 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:49.610234 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:49.610236 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.610707 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.610709 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:49.610832 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.611387 (+ 555us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:49.611396 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:49.612801 (+ 1405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:49.614619 (+ 1818us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:49.614624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:49.614626 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:49.614983 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:49.615224 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:49.615225 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:49.615226 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:49.615378 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:49.615663 (+ 285us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:49.615671 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:49.615815 (+ 144us) client_negotiation.cc:769] Sending connection context
0926 15:04:49.615925 (+ 110us) client_negotiation.cc:240] Negotiation successful
0926 15:04:49.616107 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":168,"thread_start_us":122,"threads_started":1}
I20260926 15:04:49.616986 31421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } has no permanent_uuid. Determining permanent_uuid...
Sep 26 15:04:49 dist-test-slave-jnhs krb5kdc[31190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435089, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.62@KRBTEST.COM for kudu/127.8.102.60@KRBTEST.COM
Sep 26 15:04:49 dist-test-slave-jnhs krb5kdc[31190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.60@KRBTEST.COM
Sep 26 15:04:49 dist-test-slave-jnhs krb5kdc[31190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435089, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.60@KRBTEST.COM for kudu/127.8.102.61@KRBTEST.COM
I20260926 15:04:49.623766 31273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.614567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:34327 (local address 127.0.0.1:46328)
0926 15:04:49.614621 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.614680 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:49.614720 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:49.615114 (+ 394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:49.615115 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:49.615294 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:49.615338 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:49.615340 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.616723 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.616724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:49.617390 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:49.617392 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.618001 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.618002 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:49.618065 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.618537 (+ 472us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:49.618542 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:49.620095 (+ 1553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:49.622031 (+ 1936us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:49.622033 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:49.622033 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:49.622253 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:49.622512 (+ 259us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:49.622513 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:49.622513 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:49.622581 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:49.623394 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:49.623396 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:49.623451 (+ 55us) client_negotiation.cc:769] Sending connection context
0926 15:04:49.623539 (+ 88us) client_negotiation.cc:240] Negotiation successful
0926 15:04:49.623637 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260926 15:04:49.623836 31426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.614812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46328 (local address 127.8.102.60:34327)
0926 15:04:49.614971 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.614973 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.614979 (+ 6us) server_negotiation.cc:407] Connection header received
0926 15:04:49.615006 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.615006 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.615014 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.615105 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.615430 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.616597 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.617476 (+ 879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.617927 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.617993 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.620182 (+ 2189us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:49.620190 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:49.620190 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:49.620197 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:49.621930 (+ 1733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.622350 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.622353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.622354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.622429 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.622665 (+ 236us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.622666 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.622666 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.622877 (+ 211us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:49.622972 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:49.623603 (+ 631us) server_negotiation.cc:299] Negotiation successful
0926 15:04:49.623697 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":140,"thread_start_us":97,"threads_started":1}
I20260926 15:04:49.625880 31425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.614724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46338 (local address 127.8.102.60:34327)
0926 15:04:49.614908 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.614913 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.614933 (+ 20us) server_negotiation.cc:407] Connection header received
0926 15:04:49.614956 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.614958 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.614973 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.615051 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.615946 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.616676 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.618472 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.618816 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.618886 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.622149 (+ 3263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:49.622150 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:49.622150 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:49.622156 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:49.624724 (+ 2568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.625059 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.625060 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.625060 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.625112 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.625347 (+ 235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.625348 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.625348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.625449 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:49.625617 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:49.625681 (+ 64us) server_negotiation.cc:299] Negotiation successful
0926 15:04:49.625758 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":91,"threads_started":1}
I20260926 15:04:49.625944 31271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844: Bootstrap starting.
I20260926 15:04:49.626816 31271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:49.627115 31271 log.cc:824] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:49.627959 31271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844: No bootstrap required, opened a new log
I20260926 15:04:49.628075 31423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.617310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.61:36487 (local address 127.0.0.1:46258)
0926 15:04:49.617436 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.617447 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:49.617493 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:49.617641 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:49.617642 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:49.617792 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:49.617844 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:49.617845 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.618786 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.618787 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:49.619528 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:49.619530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.620042 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.620044 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:49.620138 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.620570 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:49.620575 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:49.622024 (+ 1449us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:49.624202 (+ 2178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:49.624203 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:49.624203 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:49.624531 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:49.627378 (+ 2847us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:49.627379 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:49.627380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:49.627449 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:49.627774 (+ 325us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:49.627775 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:49.627817 (+ 42us) client_negotiation.cc:769] Sending connection context
0926 15:04:49.627888 (+ 71us) client_negotiation.cc:240] Negotiation successful
0926 15:04:49.627985 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109}
I20260926 15:04:49.628085 31350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.617387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46258 (local address 127.8.102.61:36487)
0926 15:04:49.617451 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.617453 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.617479 (+ 26us) server_negotiation.cc:407] Connection header received
0926 15:04:49.617553 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.617555 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.617565 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.617627 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.617931 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.618703 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.619624 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.619950 (+ 326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.620021 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.622115 (+ 2094us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:49.622116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:49.622117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:49.622125 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:49.623909 (+ 1784us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.624606 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.624607 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.624607 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.624665 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.627540 (+ 2875us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.627541 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.627541 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.627692 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:49.627789 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:49.627926 (+ 137us) server_negotiation.cc:299] Negotiation successful
0926 15:04:49.628000 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 15:04:49.628562 31421 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } has no permanent_uuid. Determining permanent_uuid...
I20260926 15:04:49.630785 31271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae2669468b45639a639e4fcf18b844" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } } peers { permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } } peers { permanent_uuid: "dcf4f79405a24893973978aacc782eb0" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } }
I20260926 15:04:49.630981 31271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fae2669468b45639a639e4fcf18b844, State: Initialized, Role: FOLLOWER
I20260926 15:04:49.631099 31271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae2669468b45639a639e4fcf18b844" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } } peers { permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } } peers { permanent_uuid: "dcf4f79405a24893973978aacc782eb0" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } }
I20260926 15:04:49.631489 31271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [sys.catalog]: configured and running, proceeding with master startup.
Sep 26 15:04:49 dist-test-slave-jnhs krb5kdc[31190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.62@KRBTEST.COM
I20260926 15:04:49.631441 31428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae2669468b45639a639e4fcf18b844" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } } peers { permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } } peers { permanent_uuid: "dcf4f79405a24893973978aacc782eb0" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } } }
I20260926 15:04:49.631585 31428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 15:04:49 dist-test-slave-jnhs krb5kdc[31190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435089, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.60@KRBTEST.COM for kudu/127.8.102.60@KRBTEST.COM
W20260926 15:04:49.633445 31439 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:04:49.633509 31439 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:04:49.636082 31274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.626624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59644 (local address 127.8.102.62:44829)
0926 15:04:49.626677 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.626679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.626688 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:04:49.626713 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.626714 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.626724 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.626767 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.627293 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.627987 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.628974 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.629248 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.629292 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.632508 (+ 3216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:49.632509 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:49.632510 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:49.632517 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:49.634470 (+ 1953us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.634897 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.634897 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.634898 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.634949 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.635152 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.635152 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.635153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.635275 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:49.635398 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:49.635849 (+ 451us) server_negotiation.cc:299] Negotiation successful
0926 15:04:49.635937 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260926 15:04:49.636214 31423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.628846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:34327 (local address 127.0.0.1:46344)
0926 15:04:49.628914 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.628924 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:49.628965 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:49.629075 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:49.629076 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:49.629208 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:49.629232 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:49.629233 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.629852 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.629853 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:49.630314 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:49.630315 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.630859 (+ 544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.630860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:49.630910 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.631292 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:49.631297 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:49.633026 (+ 1729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:49.635159 (+ 2133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:49.635160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:49.635160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:49.635359 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:49.635554 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:49.635554 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:49.635555 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:49.635603 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:49.635914 (+ 311us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:49.635914 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:49.635953 (+ 39us) client_negotiation.cc:769] Sending connection context
0926 15:04:49.636011 (+ 58us) client_negotiation.cc:240] Negotiation successful
0926 15:04:49.636124 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260926 15:04:49.636584 31425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.628887 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46344 (local address 127.8.102.60:34327)
0926 15:04:49.628918 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.628920 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.628952 (+ 32us) server_negotiation.cc:407] Connection header received
0926 15:04:49.629018 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.629019 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.629028 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.629075 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.629310 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.629755 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.630417 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.630767 (+ 350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.630858 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.633116 (+ 2258us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:49.633118 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:49.633118 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:49.633126 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:49.635073 (+ 1947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.635435 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.635435 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.635436 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.635485 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.635661 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.635662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.635662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.635828 (+ 166us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:49.635891 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:49.636431 (+ 540us) server_negotiation.cc:299] Negotiation successful
0926 15:04:49.636505 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13}
I20260926 15:04:49.634761 31203 master_runner.cc:186] Error getting master registration for 127.8.102.61:36487: OK, instance_id { permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" instance_seqno: 1790435089440246 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 15:04:49.637714 31203 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 15:04:49.638088 31421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0: Bootstrap starting.
I20260926 15:04:49.638856 31421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:49.639168 31421 log.cc:824] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:49.640038 31421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0: No bootstrap required, opened a new log
I20260926 15:04:49.642100 31421 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae2669468b45639a639e4fcf18b844" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } } peers { permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } } peers { permanent_uuid: "dcf4f79405a24893973978aacc782eb0" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } }
I20260926 15:04:49.642251 31421 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcf4f79405a24893973978aacc782eb0, State: Initialized, Role: FOLLOWER
I20260926 15:04:49.642343 31421 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae2669468b45639a639e4fcf18b844" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } } peers { permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } } peers { permanent_uuid: "dcf4f79405a24893973978aacc782eb0" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } }
Sep 26 15:04:49 dist-test-slave-jnhs krb5kdc[31190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.8.102.61@KRBTEST.COM
I20260926 15:04:49.642683 31442 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae2669468b45639a639e4fcf18b844" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } } peers { permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } } peers { permanent_uuid: "dcf4f79405a24893973978aacc782eb0" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } } }
I20260926 15:04:49.642694 31421 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:49.642880 31442 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:04:49.646157 31350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.638226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46262 (local address 127.8.102.61:36487)
0926 15:04:49.638265 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.638267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.638276 (+ 9us) server_negotiation.cc:407] Connection header received
0926 15:04:49.638338 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.638339 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.638348 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.638405 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.639016 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.639662 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.640404 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.640624 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.640713 (+ 89us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.643075 (+ 2362us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:49.643077 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:49.643079 (+ 2us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:49.643093 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:49.644788 (+ 1695us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.645229 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.645230 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.645230 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.645293 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.645517 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.645518 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.645519 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.645654 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:49.645806 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:49.645972 (+ 166us) server_negotiation.cc:299] Negotiation successful
0926 15:04:49.646065 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 15:04:49.647228 31453 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:04:49.647344 31453 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:04:49.647493 8600 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260926 15:04:49.648375 31245 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "6fae2669468b45639a639e4fcf18b844"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59644
I20260926 15:04:49.648504 31245 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260926 15:04:49.648548 31245 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:49.646162 31353 master_runner.cc:186] Error getting master registration for 127.8.102.61:36487: OK, instance_id { permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" instance_seqno: 1790435089440246 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260926 15:04:49.648806 31353 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260926 15:04:49.649464 31245 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae2669468b45639a639e4fcf18b844" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } } peers { permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } } peers { permanent_uuid: "dcf4f79405a24893973978aacc782eb0" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } }
I20260926 15:04:49.649675 31245 leader_election.cc:288] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [CANDIDATE]: Term 1 election: Requested vote from peers 7efd25aab3c549a89ef6360d2f68b81e (127.8.102.61:36487), dcf4f79405a24893973978aacc782eb0 (127.8.102.60:34327)
I20260926 15:04:49.650007 31395 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fae2669468b45639a639e4fcf18b844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "dcf4f79405a24893973978aacc782eb0"
I20260926 15:04:49.650133 31395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0 [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:49.651099 31395 raft_consensus.cc:2454] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fae2669468b45639a639e4fcf18b844 in term 1.
I20260926 15:04:49.651584 31219 leader_election.cc:302] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6fae2669468b45639a639e4fcf18b844, dcf4f79405a24893973978aacc782eb0; no voters:
I20260926 15:04:49.651732 31428 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [term 1 FOLLOWER]: Leader election won for term 1
I20260926 15:04:49.651908 31428 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [term 1 LEADER]: Becoming Leader. State: Replica: 6fae2669468b45639a639e4fcf18b844, State: Running, Role: LEADER
I20260926 15:04:49.651998 31428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae2669468b45639a639e4fcf18b844" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } } peers { permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } } peers { permanent_uuid: "dcf4f79405a24893973978aacc782eb0" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } }
I20260926 15:04:49.652499 31462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fae2669468b45639a639e4fcf18b844. Latest consensus state: current_term: 1 leader_uuid: "6fae2669468b45639a639e4fcf18b844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae2669468b45639a639e4fcf18b844" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } } peers { permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } } peers { permanent_uuid: "dcf4f79405a24893973978aacc782eb0" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } } }
I20260926 15:04:49.652637 31462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:49.661681 31350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.651230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46264 (local address 127.8.102.61:36487)
0926 15:04:49.651411 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.651413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.651421 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:04:49.651443 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.651444 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.651454 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.651514 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.653142 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.653891 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.654914 (+ 1023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.655681 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.655743 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.657540 (+ 1797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:49.657541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:49.657542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:49.657550 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:49.659852 (+ 2302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.660466 (+ 614us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.660468 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.660468 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.660529 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.660780 (+ 251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.660781 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.660781 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.660933 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:49.661042 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:49.661471 (+ 429us) server_negotiation.cc:299] Negotiation successful
0926 15:04:49.661567 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260926 15:04:49.661734 31425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.651434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46354 (local address 127.8.102.60:34327)
0926 15:04:49.651494 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.651496 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.651503 (+ 7us) server_negotiation.cc:407] Connection header received
0926 15:04:49.651521 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.651522 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.651531 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.651579 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.653619 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.654316 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.655277 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.655631 (+ 354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.655679 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.657654 (+ 1975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:49.657655 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:49.657656 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:49.657663 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:49.660111 (+ 2448us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.660872 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.660873 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.660873 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.660922 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.661142 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.661143 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.661144 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.661270 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:49.661349 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:49.661569 (+ 220us) server_negotiation.cc:299] Negotiation successful
0926 15:04:49.661644 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260926 15:04:49.662792 31464 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260926 15:04:49.663026 31464 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260926 15:04:49.668181 31274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.650917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59654 (local address 127.8.102.62:44829)
0926 15:04:49.650957 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.650959 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.651109 (+ 150us) server_negotiation.cc:407] Connection header received
0926 15:04:49.651220 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.651221 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.651238 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.651270 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.658040 (+ 6770us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.658867 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.663475 (+ 4608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.663698 (+ 223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.663803 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.665219 (+ 1416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:49.665221 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:49.665221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:49.665229 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:49.666915 (+ 1686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.667329 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.667330 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.667331 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.667388 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.667595 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.667596 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.667596 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.667751 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:49.667844 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:49.667987 (+ 143us) server_negotiation.cc:299] Negotiation successful
0926 15:04:49.668063 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260926 15:04:49.675617 31346 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } attempt: 3
I20260926 15:04:49.675722 31395 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0 [term 1 FOLLOWER]: Refusing update from remote peer 6fae2669468b45639a639e4fcf18b844: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:04:49.676126 31462 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dcf4f79405a24893973978aacc782eb0" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:04:49.677937 31442 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fae2669468b45639a639e4fcf18b844. Latest consensus state: current_term: 1 leader_uuid: "6fae2669468b45639a639e4fcf18b844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae2669468b45639a639e4fcf18b844" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } } peers { permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } } peers { permanent_uuid: "dcf4f79405a24893973978aacc782eb0" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } } }
I20260926 15:04:49.678031 31442 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0 [sys.catalog]: This master's current role is: FOLLOWER
Sep 26 15:04:49 dist-test-slave-jnhs krb5kdc[31190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790435089, etypes {rep=17 tkt=17 ses=17}, kudu/127.8.102.61@KRBTEST.COM for kudu/127.8.102.60@KRBTEST.COM
I20260926 15:04:49.683568 31462 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fae2669468b45639a639e4fcf18b844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae2669468b45639a639e4fcf18b844" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } } peers { permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } } peers { permanent_uuid: "dcf4f79405a24893973978aacc782eb0" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } } }
I20260926 15:04:49.683681 31462 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:49.683772 31465 mvcc.cc:205] Tried to move back new op lower bound from 7333622127308759040 to 7333622127215157248. Current Snapshot: MvccSnapshot[applied={T|T < 7333622127308759040}]
I20260926 15:04:49.684276 31348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.675906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.8.102.60:34327 (local address 127.0.0.1:46366)
0926 15:04:49.676005 (+ 99us) negotiation.cc:107] Waiting for socket to connect
0926 15:04:49.676018 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0926 15:04:49.676092 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0926 15:04:49.676311 (+ 219us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0926 15:04:49.676312 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0926 15:04:49.676506 (+ 194us) client_negotiation.cc:189] Negotiated authn=SASL
0926 15:04:49.676565 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:49.676567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.677518 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.677519 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:49.678302 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0926 15:04:49.678305 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.678835 (+ 530us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.678836 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0926 15:04:49.678904 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.679385 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0926 15:04:49.679390 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0926 15:04:49.681087 (+ 1697us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0926 15:04:49.683002 (+ 1915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:49.683003 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:49.683004 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:49.683226 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:49.683436 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0926 15:04:49.683436 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0926 15:04:49.683437 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0926 15:04:49.683499 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0926 15:04:49.683950 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0926 15:04:49.683951 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0926 15:04:49.684001 (+ 50us) client_negotiation.cc:769] Sending connection context
0926 15:04:49.684066 (+ 65us) client_negotiation.cc:240] Negotiation successful
0926 15:04:49.684177 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260926 15:04:49.684826 31425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:49.675996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46366 (local address 127.8.102.60:34327)
0926 15:04:49.676040 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.676042 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.676081 (+ 39us) server_negotiation.cc:407] Connection header received
0926 15:04:49.676156 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.676157 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.676171 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.676240 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.676749 (+ 509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.677418 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.678393 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:49.678768 (+ 375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:49.678832 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:49.681162 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0926 15:04:49.681163 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0926 15:04:49.681164 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0926 15:04:49.681171 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0926 15:04:49.682909 (+ 1738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.683307 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.683308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.683308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.683368 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0926 15:04:49.683570 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0926 15:04:49.683571 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0926 15:04:49.683571 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0926 15:04:49.683812 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0926 15:04:49.683920 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:49.684655 (+ 735us) server_negotiation.cc:299] Negotiation successful
0926 15:04:49.684750 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260926 15:04:49.684922 31442 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fae2669468b45639a639e4fcf18b844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae2669468b45639a639e4fcf18b844" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } } peers { permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } } peers { permanent_uuid: "dcf4f79405a24893973978aacc782eb0" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } } }
I20260926 15:04:49.685129 31442 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0 [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:04:49.685892 31466 mvcc.cc:205] Tried to move back new op lower bound from 7333622127308759040 to 7333622127215157248. Current Snapshot: MvccSnapshot[applied={T|T < 7333622127308759040}]
I20260926 15:04:49.686497 31464 catalog_manager.cc:1347] Generated new cluster ID: ee4a1790d5c941028b5015ead6b11d83
I20260926 15:04:49.686551 31464 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260926 15:04:49.686560 31346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e: Bootstrap starting.
I20260926 15:04:49.687315 31346 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e: Neither blocks nor log segments found. Creating new log.
I20260926 15:04:49.687613 31346 log.cc:824] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e: Log is configured to *not* fsync() on all Append() calls
I20260926 15:04:49.688464 31346 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e: No bootstrap required, opened a new log
I20260926 15:04:49.690498 31346 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae2669468b45639a639e4fcf18b844" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } } peers { permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } } peers { permanent_uuid: "dcf4f79405a24893973978aacc782eb0" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } }
I20260926 15:04:49.690677 31346 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7efd25aab3c549a89ef6360d2f68b81e, State: Initialized, Role: FOLLOWER
I20260926 15:04:49.690824 31346 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae2669468b45639a639e4fcf18b844" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } } peers { permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } } peers { permanent_uuid: "dcf4f79405a24893973978aacc782eb0" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } }
I20260926 15:04:49.691174 31346 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e [sys.catalog]: configured and running, proceeding with master startup.
I20260926 15:04:49.691270 31320 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6fae2669468b45639a639e4fcf18b844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "7efd25aab3c549a89ef6360d2f68b81e"
I20260926 15:04:49.691375 31320 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e [term 0 FOLLOWER]: Advancing to term 1
I20260926 15:04:49.692322 31320 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6fae2669468b45639a639e4fcf18b844 in term 1.
I20260926 15:04:49.692521 31319 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e [term 1 FOLLOWER]: Refusing update from remote peer 6fae2669468b45639a639e4fcf18b844: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260926 15:04:49.693123 31428 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260926 15:04:49.691110 31471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae2669468b45639a639e4fcf18b844" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } } peers { permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } } peers { permanent_uuid: "dcf4f79405a24893973978aacc782eb0" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } } }
I20260926 15:04:49.693441 31471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:04:49.695155 31471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fae2669468b45639a639e4fcf18b844. Latest consensus state: current_term: 1 leader_uuid: "6fae2669468b45639a639e4fcf18b844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae2669468b45639a639e4fcf18b844" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } } peers { permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } } peers { permanent_uuid: "dcf4f79405a24893973978aacc782eb0" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } } }
I20260926 15:04:49.695243 31471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e [sys.catalog]: This master's current role is: FOLLOWER
I20260926 15:04:49.695274 31471 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "6fae2669468b45639a639e4fcf18b844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae2669468b45639a639e4fcf18b844" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } } peers { permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } } peers { permanent_uuid: "dcf4f79405a24893973978aacc782eb0" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } } }
I20260926 15:04:49.695312 31471 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e [sys.catalog]: This master's current role is: FOLLOWER
W20260926 15:04:49.695894 31482 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260926 15:04:49.695991 31482 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260926 15:04:49.711470 31428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "6fae2669468b45639a639e4fcf18b844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6fae2669468b45639a639e4fcf18b844" member_type: VOTER last_known_addr { host: "127.8.102.62" port: 44829 } } peers { permanent_uuid: "7efd25aab3c549a89ef6360d2f68b81e" member_type: VOTER last_known_addr { host: "127.8.102.61" port: 36487 } } peers { permanent_uuid: "dcf4f79405a24893973978aacc782eb0" member_type: VOTER last_known_addr { host: "127.8.102.60" port: 34327 } } }
I20260926 15:04:49.711611 31428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844 [sys.catalog]: This master's current role is: LEADER
I20260926 15:04:49.714758 31464 catalog_manager.cc:1370] Generated new certificate authority record
I20260926 15:04:49.715581 31464 catalog_manager.cc:1504] Loading token signing keys...
I20260926 15:04:49.725391 31464 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fae2669468b45639a639e4fcf18b844: Generated new TSK 0
I20260926 15:04:49.725630 31464 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260926 15:04:49.755517 31274 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:49.753779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59658 (local address 127.8.102.62:44829)
0926 15:04:49.753831 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.753834 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.754821 (+ 987us) server_negotiation.cc:407] Connection header received
0926 15:04:49.754930 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.754931 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.754944 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.754975 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.755434 (+ 459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59658: BlockingRecv error: recv got EOF from 127.0.0.1:59658 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260926 15:04:49.756322 31350 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:49.754258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46278 (local address 127.8.102.61:36487)
0926 15:04:49.754298 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.754300 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.755765 (+ 1465us) server_negotiation.cc:407] Connection header received
0926 15:04:49.755866 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.755867 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.755877 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.755906 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.756260 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46278: BlockingRecv error: recv got EOF from 127.0.0.1:46278 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 15:04:49.757033 31425 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:49.754539 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46378 (local address 127.8.102.60:34327)
0926 15:04:49.754578 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.754580 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.756540 (+ 1960us) server_negotiation.cc:407] Connection header received
0926 15:04:49.756636 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.756637 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.756649 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.756677 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.756973 (+ 296us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46378: BlockingRecv error: recv got EOF from 127.0.0.1:46378 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260926 15:04:49.759640 31350 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:49.758579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46284 (local address 127.8.102.61:36487)
0926 15:04:49.758617 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.758618 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.758861 (+ 243us) server_negotiation.cc:407] Connection header received
0926 15:04:49.758976 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.758977 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.758996 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.759026 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.759590 (+ 564us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46284: BlockingRecv error: recv got EOF from 127.0.0.1:46284 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
W20260926 15:04:49.762779 31350 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:49.762067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46296 (local address 127.8.102.61:36487)
0926 15:04:49.762113 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.762115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.762227 (+ 112us) server_negotiation.cc:407] Connection header received
0926 15:04:49.762324 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.762325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.762335 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.762364 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.762730 (+ 366us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46296: BlockingRecv error: recv got EOF from 127.0.0.1:46296 (error 108)
Metrics: {"server-negotiator.queue_time_us":27}
W20260926 15:04:49.768028 31350 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:49.766020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46306 (local address 127.8.102.61:36487)
0926 15:04:49.766064 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.766067 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.767418 (+ 1351us) server_negotiation.cc:407] Connection header received
0926 15:04:49.767527 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.767528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.767540 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.767569 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.767969 (+ 400us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46306: BlockingRecv error: recv got EOF from 127.0.0.1:46306 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 15:04:49.774165 31350 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:49.773326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46320 (local address 127.8.102.61:36487)
0926 15:04:49.773366 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.773368 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.773592 (+ 224us) server_negotiation.cc:407] Connection header received
0926 15:04:49.773706 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.773707 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.773718 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.773750 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.774106 (+ 356us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46320: BlockingRecv error: recv got EOF from 127.0.0.1:46320 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260926 15:04:49.784596 31350 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:49.783620 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46336 (local address 127.8.102.61:36487)
0926 15:04:49.783666 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.783669 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.783955 (+ 286us) server_negotiation.cc:407] Connection header received
0926 15:04:49.784068 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.784069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.784082 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.784113 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.784535 (+ 422us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46336: BlockingRecv error: recv got EOF from 127.0.0.1:46336 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 15:04:49.803299 31350 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:49.802128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46346 (local address 127.8.102.61:36487)
0926 15:04:49.802176 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.802179 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.802598 (+ 419us) server_negotiation.cc:407] Connection header received
0926 15:04:49.802756 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.802757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.802770 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.802808 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.803238 (+ 430us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46346: BlockingRecv error: recv got EOF from 127.0.0.1:46346 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260926 15:04:49.837745 31350 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:49.836788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46352 (local address 127.8.102.61:36487)
0926 15:04:49.836839 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.836841 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.837130 (+ 289us) server_negotiation.cc:407] Connection header received
0926 15:04:49.837233 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.837234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.837246 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.837275 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.837683 (+ 408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46352: BlockingRecv error: recv got EOF from 127.0.0.1:46352 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260926 15:04:49.904127 31350 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:49.903305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46362 (local address 127.8.102.61:36487)
0926 15:04:49.903381 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:49.903384 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:49.903505 (+ 121us) server_negotiation.cc:407] Connection header received
0926 15:04:49.903577 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:49.903578 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:49.903589 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:49.903633 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:49.904061 (+ 428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46362: BlockingRecv error: recv got EOF from 127.0.0.1:46362 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
W20260926 15:04:50.034030 31350 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:50.033269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46366 (local address 127.8.102.61:36487)
0926 15:04:50.033339 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:50.033341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:50.033506 (+ 165us) server_negotiation.cc:407] Connection header received
0926 15:04:50.033544 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:50.033545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:50.033556 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:50.033607 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:50.033944 (+ 337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46366: BlockingRecv error: recv got EOF from 127.0.0.1:46366 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260926 15:04:50.292340 31350 negotiation.cc:337] Failed RPC negotiation. Trace:
0926 15:04:50.291448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46374 (local address 127.8.102.61:36487)
0926 15:04:50.291530 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:50.291532 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:50.291590 (+ 58us) server_negotiation.cc:407] Connection header received
0926 15:04:50.291660 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:50.291661 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:50.291673 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:50.291730 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0926 15:04:50.292259 (+ 529us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46374: BlockingRecv error: recv got EOF from 127.0.0.1:46374 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260926 15:04:50.647846 31453 catalog_manager.cc:1259] Loaded cluster ID: ee4a1790d5c941028b5015ead6b11d83
I20260926 15:04:50.647924 31453 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0: loading cluster ID for follower catalog manager: success
I20260926 15:04:50.648813 31453 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0: acquiring CA information for follower catalog manager: success
I20260926 15:04:50.649138 31453 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dcf4f79405a24893973978aacc782eb0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 15:04:50.696518 31482 catalog_manager.cc:1259] Loaded cluster ID: ee4a1790d5c941028b5015ead6b11d83
I20260926 15:04:50.696600 31482 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e: loading cluster ID for follower catalog manager: success
I20260926 15:04:50.697362 31482 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e: acquiring CA information for follower catalog manager: success
I20260926 15:04:50.697630 31482 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7efd25aab3c549a89ef6360d2f68b81e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260926 15:04:50.808359 31553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:50.805567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46382 (local address 127.8.102.61:36487)
0926 15:04:50.805711 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:50.805713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:50.805829 (+ 116us) server_negotiation.cc:407] Connection header received
0926 15:04:50.805863 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:50.805864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:50.805914 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:50.806001 (+ 87us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:04:50.806190 (+ 189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:50.806784 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:50.807560 (+ 776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:50.807760 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:50.807828 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:50.808015 (+ 187us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:04:50.808155 (+ 140us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:04:50.808219 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:50.808264 (+ 45us) server_negotiation.cc:299] Negotiation successful
0926 15:04:50.808283 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":86,"threads_started":1}
I20260926 15:04:50.812687 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:50.810222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46388 (local address 127.8.102.60:34327)
0926 15:04:50.810349 (+ 127us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:50.810351 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:50.810359 (+ 8us) server_negotiation.cc:407] Connection header received
0926 15:04:50.810395 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:50.810396 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:50.810430 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:50.810503 (+ 73us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:04:50.810668 (+ 165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:50.811171 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:50.811934 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:50.812173 (+ 239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:50.812239 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:50.812387 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:04:50.812503 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:04:50.812566 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:50.812604 (+ 38us) server_negotiation.cc:299] Negotiation successful
0926 15:04:50.812624 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":69,"threads_started":1}
I20260926 15:04:50.817452 31565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:50.814412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59668 (local address 127.8.102.62:44829)
0926 15:04:50.814535 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:50.814538 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:50.814603 (+ 65us) server_negotiation.cc:407] Connection header received
0926 15:04:50.814689 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:50.814690 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:50.814743 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:50.814826 (+ 83us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:04:50.815019 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:50.815603 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:50.816505 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:50.816829 (+ 324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:50.816895 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:50.817051 (+ 156us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:04:50.817144 (+ 93us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:04:50.817229 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:50.817327 (+ 98us) server_negotiation.cc:299] Negotiation successful
0926 15:04:50.817357 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":52,"threads_started":1}
I20260926 15:04:50.817724 31559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:50.814835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46402 (local address 127.8.102.60:34327)
0926 15:04:50.814879 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:50.814881 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:50.814971 (+ 90us) server_negotiation.cc:407] Connection header received
0926 15:04:50.815016 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:50.815017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:50.815032 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:50.815086 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:04:50.815226 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:50.815933 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:50.816871 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:50.817183 (+ 312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:50.817221 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:50.817454 (+ 233us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:04:50.817488 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:04:50.817537 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:50.817636 (+ 99us) server_negotiation.cc:299] Negotiation successful
0926 15:04:50.817657 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260926 15:04:50.817858 31553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0926 15:04:50.814532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46396 (local address 127.8.102.61:36487)
0926 15:04:50.814587 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0926 15:04:50.814589 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0926 15:04:50.814730 (+ 141us) server_negotiation.cc:407] Connection header received
0926 15:04:50.814845 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0926 15:04:50.814846 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0926 15:04:50.814858 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0926 15:04:50.814909 (+ 51us) server_negotiation.cc:226] Negotiated authn=TOKEN
0926 15:04:50.815097 (+ 188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:50.815876 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:50.816822 (+ 946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0926 15:04:50.817126 (+ 304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0926 15:04:50.817191 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0926 15:04:50.817346 (+ 155us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0926 15:04:50.817382 (+ 36us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0926 15:04:50.817431 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0926 15:04:50.817752 (+ 321us) server_negotiation.cc:299] Negotiation successful
0926 15:04:50.817779 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260926 15:04:50.823933 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 31203
I20260926 15:04:50.829452 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 31278
I20260926 15:04:50.835124 8600 external_mini_cluster.cc:1595] Killing /tmp/dist-test-task17_aQt/build/release/bin/kudu with pid 31353
2026-09-26T15:04:50Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2855 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5807 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (280479 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST